US009116633B2
`
`US 9,116,633 B2
`(10) Patent No.:
`a2) United States Patent
`Sancheti et al.
`(45) Date of Patent:
`Aug, 25, 2015
`
`
`(54) INFORMATION MANAGEMENT OF
`VIRTUAL MACHINES HAVING MAPPED
`STORAGE DEVICES
`
`USPC. vecccsceceee 711/6, 203, 209; 707/640, 645, 649,
`707/653, 661, 667, 674; 718/1, 104
`See application file for complete search history.
`
`(71)
`
`Applicant: CommVault Systems, Inc., Oceanport,
`NJ (US)
`
`(56)
`
`References Cited
`
`(72)
`
`Inventors:
`
`Ashwin Gautamchand Sancheti, Ocean
`City, NJ (US): Rahul S. Pawar,
`Marlboro, NJ (US)
`
`(73)
`
`Assignee: Commvault Systems, Inc., Tinton Falls,
`NI (US)
`
`(*)
`
`Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 264 days.
`
`EP
`EP
`
`U.S. PATENT DOCUMENTS
`
`4,686,620 A
`4,995,035 A
`
`8/1987
`2/1991
`
`Ng
`Cole etal.
`
`(Continued)
`
`FOREIGN PATENT DOCUMENTS
`
`0259912 Al
`0405926 A2
`
`3/1988
`1/1991
`
`(Continued)
`
`OTHER PUBLICATIONS
`
`(21)
`
`Appl. No.: 13/790,226
`
`(22)
`
`Filed:
`
`Mar. 8, 2013
`
`(65)
`
`Prior Publication Data
`
`US 2013/0262801 Al
`
`Oct. 3, 2013
`
`Related U.S. Application Data
`
`(60)
`
`(51)
`
`(52)
`
`(58)
`
`Provisional application No. 61/618,663, filed on Mar.
`30, 2012.
`
`(2006.01)
`(2006.01)
`(2006.01)
`(2006.01)
`
`Int. Cl.
`GO6F 12/00
`GO6F 3/06
`GO6F 9/455
`GOOF 12/12
`U.S.CL
`CPC .ee GO6F 3/065 (2013.01); GO6F 9/455
`(2013.01); GO6F 9/45558 (2013.01); GO6F
`12/12 (2013.01); GO6F 12/127 (2013.01);
`GO6F 2009/45562 (2013.01)
`Field of Classification Search
`CPC .…… GO6P 3/065; GO6E 9/455; GO6E 9/4558;
`GO6F 12/12; GO6F 12/127
`
`ee
`
`a ee
`
`physical machine A
`(sop H+ 1008
`1078
`
`
`<
`
`tos
`[ar
`
`401b
`
`
`er |
`
`production
`data store
`
`
`
`+1098
`
`US. Appl. No. 13/765,389, filed Feb. 12, 2013, Kripalani.
`(Continued)
`
`Primary Examiner — Reba | Elmore
`(74) Attorney, Agent, or Firm — Perkins Coi LLP
`
`(57)
`
`ABSTRACT
`
`Software, firmware, and systems are described herein that
`create and use a non-production copy of a virtual machine for
`reverting or restoring the
`virtual machine. The virtual
`machine is associated with an external storage device via a
`logical mapping. A snapshot is taken of a virtual disk associ-
`ated with the virtual machine to create a snapshot copy of the
`virtual disk. A snapshot is taken of at least a portion of the
`mapped external storage device to create a snapshot copy of
`the mapped external storage device. The snapshot copy of the
`virtual disk is associated with the snapshot copy of the
`mapped external storage device. The snapshot copies can then
`be used to either revert or restore the virtual machine.
`
`20 Claims, 12 Drawing Sheets
`
`[108
`
`109
`
`108) virtualmachinahost 1052 1
`
`1
`virtual machine 140a
`1
`1
`1
`1
`1)
`|
`
`T1158
`
`7 1200
`
`Tsb
`下 tb
`
`1
`1
`1
`
`
`mai
`
`i
`
`machine
`
`
`
`
`
`
`
`
`
`Lyra machine host 1050
`
`WIZ, Inc. EXHIBIT - 1020
`WIZ, Inc. v. Orca Security LTD. - IPR2024-00220
`
`
`
`
`WIZ, Inc. EXHIBIT - 1020
`WIZ, Inc. v. Orca Security LTD.
`
`

`

`US 9,116,633 B2
`Page 2
`
`
`7,076,270 B2
`7,219,162 B2
`7,246,207 B2
`7,395,282 Bl
`7,448,079 B2
`7,552,279 BL*
`7,631,351 B2
`7,730,035 B2
`7,756,835 B2
`7,756,964 B2
`7,793,307 B2*
`7,840,537 B2
`7,882,077 B2
`7,899,788 B2
`7,937,421 B2
`8,069,271 B2
`8,140,786 B2
`8,219,524 B2
`8,307,177 B2
`8,578,126 BL*
`2004/0030668 Al*
`2004/0030822 Al*
`2006/0224846 Al
`2007/0100792 AL*
`2007/0203938 Al
`2007/0234302 Al*
`2008/0189468 Al*
`2008/0244177 Al*
`2008/0313371 Al*
`2009/0144416 Al
`2009/0222496 Al*
`2009/0300023 Al*
`2009/0319534 Al
`2009/0327471 Al
`2009/0327477 Al*
`2010/0070725 Al*
`2010/0161919 Al*
`2010/0242096 Al
`2010/0332454 Al
`2011/0087632 Al
`2011/0093471 Al
`2011/0179414 AL*
`2011/0219144 Al*
`2012/0084262 Al
`2012/0254364 Al
`2013/0054533 Al*
`2013/0061014 Al
`
`Jaggers et al.
`7/2006
`5/2007 Donker et al.
`7/2007
`Kottomtharayil et al.
`7/2008
`Crescenti et al.
`11/2008
`Tremain
`6/2009
`Gandler oo... 711/114
`“12/2009
`Erofeev
`6/2010
`Berger et al.
`7/2010
`Pugh
`7/2010
`Madison, Jr. et al.
`9/2010
`Gokhale etal. .............. 719/324
`11/2010
`Gokhale et al.
`2/2011
`Gokhale et al.
`3/2011
`Chandhok et al.
`5/2011
`Mikesell et al.
`11/2011
`Brunet et al.
`3/2012
`Bunte et al.
`7/2012
`Gokhale
`11/2012
`Prahlad et al.
`11/2013 Gaonkar et al 711/201
`2/2004 Pawlowski et al.
`. 707/1
`2/2004
`Rajanetal. wc
`ee 711/4
`10/2006 Amarendran et al.
`5/2007
`Lent et al. wee 707/1
`8/2007 Prahlad et al.
`10/2007
`Suzuki et al. oc. 717/126
`Schmidt et al. .... 711/6
`8/2008
`
`.711114
`10/2008
`Crescenti et al.
`..
`12/2008 Kedem etal. ..
`.. 710/68
`6/2009
`Chatley et al.
`9/2009
`Liu et al. woe 707/204
`12/2009
`Vaghani .pet 707/10
`12/2009
`Gokhale
`12/2009
`Astete et al.
`
`. 709/224
`12/2009
`Madison etal. ...
`. TLL/162
`3/2010
`Prahlad etal.
`.....
`6/2010 Dodgson et al 711161
`9/2010
`Varadharajan et al.
`12/2010 Prahlad et al.
`4/2011
`Subramanian et al.
`4/2011
`Brockway et al.
`7/2011
`Gogginetal. wee 718/1
`92011
`Amitetal. wo. 709/247
`4/2012 Dwarampudi et al.
`10/2012
`Vijayan
`2/2013
`再 ao et ae 707/649
`3/2013 Prahlad et al.
`
`
`
`FOREIGN PATENT DOCUMENTS
`
`EP
`EP
`EP
`EP
`EP
`WO
`WO
`
`0467546 A2
`0774715 Al
`0809184 Al
`0899662 Al
`0981090 Al
`WO-9513580 Al
`WO-9912098 Al
`
`1/1992
`5/1997
`11/1997
`3/1999
`2/2000
`5/1995
`3/1999
`
`OTHER PUBLICATIONS
`
`U.S. Appl. No. 13/789,871, filed Mar. 8, 2013, Kumarasamy et al.
`US. Appl. No. 13/790,028, filed Mar. 8, 2013, Kumarasamy et al.
`USS. Appl. No. 13/926,332, filed Jun. 25, 2013, Dwarampudi et al.
`Armstead et
`al., “Implementation of a Campus-wide Distributed
`Mass Storage Service: The Dream vs. Reality,” JEEE, Sep. 11-14,
`1995, pp. 190-199.
`Arneson, “Mass Storage Archiving in Network Environments,”
`Digest of Papers, Ninth IEEE Symposium on Mass Storage Systems,
`Oct. 31, 1988-Nov. 3, 1988, pp. 45-50, Monterey, CA.
`Brandon, J., “Virtualization Shakes Up Backup Strategy,” <http://
`www.computerworld.com>, internet accessed on Mar. 6, 2008, 3
`pages.
`Cabrera et al., “ADSM: A Multi-Platform, Scalable, Backup and
`Archive Mass Storage System,” Digest of Papers, Compcon “95,
`Proceedings of the 40th IEEE Computer Society International Con-
`ference, Mar. 5, 1995-Mar. 9, 1995, pp. 420-427, San Francisco, CA.
`CommVault Systems, Inc., “A CommVault White Paper: VMware
`
`
`
`
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`4/1991
`3/1992
`T1992
`3/1993
`5/1993
`7/1993
`8/1993
`8/1993
`8/1993
`1/1994
`1/1994
`2/1994
`6/1994
`7/1994
`9/1994
`4/1995
`9/1995
`2/1996
`2/1996
`4/1996
`8/1996
`8/1996
`9/1996
`4/1997
`6/1997
`9/1997
`12/1997
`3/1998
`5/1998
`5/1998
`6/1998
`6/1998
`7/1998
`9/1998
`9/1998
`9/1998
`2/1999
`3/1999
`5/1999
`7/1999
`9/1999
`10/1999
`2/2000
`2/2000
`4/2000
`6/2000
`7/2000
`10/2000
`10/2000
`11/2000
`11/2000
`12/2000
`12/2000
`4/2001
`7/2001
`7/2001
`8/2001
`10/2001
`11/2001
`12/2001
`12/2001
`12/2001
`1/2002
`3/2002
`3/2002
`5/2002
`5/2002
`7/2002
`11/2002
`2/2003
`3/2003
`§/2003
`6/2003
`“12/2003
`4/2006
`
`Griffin et al.
`Dong et al.
`Cheffetz et al.
`Kitajima et al.
`Masters
`Nakano et al.
`Anglin et al.
`Eastridge et al.
`Eastridge et al.
`Fortier et al.
`Kenley et al.
`Stoppani, Jr.
`Rogan et al.
`Saether et al.
`Flynn et al.
`Fecteau et al.
`Hayashi
`Allen
`Pisello et al.
`Martin et al.
`Carpenter et al.
`Yanai et al.
`Balk
`Crockett et al.
`Dunphy et al.
`Huai et al.
`Ding et al.
`Squibb
`Kullick et al.
`Saxon
`Senator et al.
`Crouse et al.
`Whiting et al.
`Nielsen
`Johnson et al.
`Mortis
`Blumenau
`Ebrahim
`Ofek
`Perks
`Aviani, Jr.
`Beeler, Jr.
`Cannon et al.
`Anglin
`Ulrich et al.
`Kedem
`Ying
`Low et al.
`Sidwell
`Cannon et al.
`Urevig et al.
`Mutalik et al.
`Yeager
`Barney et al.
`Anglin
`Dunham
`Vahalia et al.
`Aoyama et al.
`Xu et al.
`Long
`Crighton
`Carteau
`Hubis et al.
`Eastridge et al.
`Goodman et al.
`Pothapragada et al.
`Devine et al. wee 718/1
`Blumenau et al.
`Ofek et al.
`Devireddy et al.
`Lagueux, Jr. et al.
`O’Connor
`Ching et al.
`Nguyen et al.
`Crescenti et al.
`
`PRE EEE EES ESE EEE SESE ESE EEE EEE EEL EEE ES ES ESE EEE EEE EE EE ES
`
`Bl
`Bl
`Bl
`Bl
`B1
`B1
`B1
`Bl
`Bl
`Bl
`E
`Bl
`Bl
`
`Bl
`Bl
`B2
`Bl
`Bl
`Bl
`B2
`Bl
`
`5,005,122
`5,093,912
`5,133,065
`5,193,154
`5,212,772
`5,226,157
`5,239,647
`5,241,668
`5,241,670
`5,276,860
`5,276,867
`5,287,500
`5,321,816
`5,333,315
`5,347,653
`5,410,700
`5,448,724
`5,491,810
`5,495,607
`5,504,873
`5,544,345
`5,544,347
`5,559,957
`5,619,644
`5,638,509
`5,673,381
`5,699,361
`5,729,743
`5,751,997
`5,758,359
`5,761,677
`5,764,972
`5,778,395
`5,812,398
`5,813,009
`5,813,017
`5,875,478
`5,887,134
`5,901,327
`5,924,102
`5,950,205
`5,974,563
`6,021,415
`6,026,414
`6,052,735
`6,076,148
`6,094,416
`6,131,095
`6,131,190
`6,148,412
`6,154,787
`6,161,111
`6,167,402
`6,212,512
`6,260,069
`6,269,431
`6,275,953
`6,301,592
`6,324,581
`6,328,766
`6,330,570
`6,330,642
`6,343,324
`RE37,601
`6,356,801
`6,389,432
`6,397,242
`6,421,711
`6,487,561
`6,519,679
`6,538,669
`6,564,228
`6,581,076
`6,658,526
`7,035,880
`
`

`

`US 9,116,633 B2
`Page 3
`
`
`
`(56)
`
`References Cited
`
`OTHER PUBLICATIONS
`
`Consolidated Backup (VCB) Certification Information Kit,” 2007,
`23 pages.
`Inc., “CommVault Solutions—VMware,”
`CommVault Systems,
`<http://www.commvault.com/solutions/vmware/>,
`internet
`accessed Mar. 24, 2008, 2 pages.
`CommVault Systems, Inc., “Enhanced Protection and Manageability
`of Virtual Servers,” Partner Solution Brief, 2008, 6 pages.
`Davis, D., “3 VMware Consolidated Backup (VCB) Utilities You
`Should Know.” Petri
`IT Knowlegebase, <http://www.petri.co.il/
`vmware-consolidated-backup-utilities.htm>, internet accessed on
`Jul. 14, 2008, 7 pages.
`Davis, D., “Understanding VMware VMX Configuration Files,” Petri
`
`IT
`Knowledgebase, — <http:/www.petri.co.il/virtual_vmware_
`vmx_ configuration files.htm>, internet accessed on Jun. 19, 2008,
`6 pages.
`Davis, D., “WMware Server & Workstation Disk Files Explained,”
`Petri IT Knowledgebase, <http://www.petri.co.il/virtual_vmware_
`files_explained.htm>, internet accessed on Jun. 19, 2008, 5 pages.
`Davis, D., “VMware Versions
`Compared.” Petri IT Knowledgebase,
`<http://www.petri.co.il/virtual_vmware__versions__compared.
`htm>, internet accessed on Apr. 28, 2008, 6 pages.
`Eitel, “Backup and Storage Management in Distributed Heteroge-
`neous Environments,” JEEE, Jun. 12-16, 1994, pp. 124-126.
`International Preliminary Report on Patentability and Written Opin-
`ion for PCT/US2011/054374, dated Apr. 11, 2013, 6 pages.
`International Search Report and Written Opinion for PCT/US2011/
`054374, dated May 2, 2012, 9 pages.
`Jander, M., “Launching Storage-Area Net,’ Data Communications,
`US, McGraw Hill, NY, vol. 27, No. 4 (Mar. 21, 1998), pp. 64-72.
`Jason Gait, “The Optical File
`Cabinet:
`A Random-Access File Sys-
`tem for Write-Once Optical Disks,” JEEE Computer, vol. 21, No. 6,
`pp. 11-22 (Jun. 1988).
`Microsoft Corporation, “How NTFS Works,” Windows Server
`TechCenter, updated Mar. 28, 2003, internet accessed Mar. 26, 2008,
`26 pages.
`Rosenblum et al., “The Design and Implementation of a Log-Struc-
`tured File System,” Operating Systems Review SIGOPS, vol. 25, No.
`5, New York, US, pp. 1-15 (May 1991).
`Sanbarrow.com,
`“Disktype-table.”
`<http://sanbarrow.com/vmdk/
`disktypes.html>, internet accessed on Jul. 22, 2008, 4 pages.
`Sanbarrow.com, “Files Used by a VM,” <http://sanbarrow.com/vmx/
`vmx-files-used-by-avm.html>, internet accessed on Jul. 22, 2008, 2
`pages.
`Sanbarrow.com, “Monolithic Versus Split Disks,” <http://sanbarrow.
`com/vmdk/monolithicversusspllit-html>, internet accessed on Jul.
`14, 2008, 2 pages.
`VMware, Inc., “Open Virtual Machine Format,” <http://www.
`vmware.com/appliances/learn/ovf.html>, internet accessed on May
`6, 2008, 2 pages.
`VMware, Inc., “OVF, Open Virtual Machine Format Specification,
`version 0.9,” White Paper, <http://www.vmware.com>, 2007, 50
`pages.
`VMware, Inc., “The Open Virtual Machine Format Whitepaper for
`OVF Specification, version 0.9,” White Paper, <http:/‘www.vmware.
`com>, 2007, 16 pages.
`VMware, Inc., “Understanding VMware Consolidated Backup,”
`White Paper, <http://www.vmware.com>, 2007, 11 pages.
`
`VMware, Inc., “Using VMware Infrastructure for Backup and
`Restore.” Best
`Practices, <http://www.vmware.com>, 2006, 20
`pages.
`VMware, Inc., “Virtual Disk API Programming Guide,” <http://
`www.vmware.com>, Revision Apr. 11, 2008, 2008, 44 pages.
`VMware, Inc., “Virtual Disk Format 1.1,’ VMware Technical Note,
`<http://www.vmware.com>, Revision Nov. 13, 2007, Version 1.1,
`2007, 18 pages.
`VMware, Inc., “Virtual Machine Backup Guide, ESX Server 3.0.1
`and VirtualCenter 2.0.1,” <http://www.vmware.com>, updated Nov.
`21, 2007, 74 pages.
`VMware, Inc., “Virtual Machine Backup Guide, ESX Server 3.5,
`ESX Server 3i version 3.5, VirtualCenter 2.5,” <http://www.vmware.
`com>, updated Feb. 21, 2008, 78 pages.
`VMware, Inc., “Virtualized iSCSI Sans: Flexible, Scalable Enter-
`prise Storage for Virtual Infrastructures,” White Paper, <http://www.
`vmware.com>, Mar. 2008, 13 pages.
`VMware, Inc., “VMware Consolidated Backup, Improvements in
`Version 3.5,” Information Guide, <http://www.vmware.com>, 2007,
`11 pages.
`VMware, Inc., “WMware Consolidated Backup,” Product Datasheet,
`<http://www.vmware.com>, 2007, 2 pages.
`VMware, Inc.,““WMware ESX 3.5,” Product Datasheet, <http://www.
`vmware.com>, 2008, 4 pages.
`VMware, Inc., “VMware GSX Server 3.2, Disk Types: Virtual and
`Physical,”
`<http://www.vmware.com/support/gsx3/doc/disks
`types__gsx.html>, internet accessed on Mar. 25, 2008, 2 pages.
`VMware, Inc., “WMware OVF Tool,” Technical Note, <http://www.
`vmware.com>, 2007, 4 pages.
`a Linear
`VMware, Inc., “VMware Workstation 5.0, Snapshots in
`Process,”
`
`<http:/www.vmware.com/ support’ ws5/doc/ws__pre-
`serve_sshot_linear.html>, internet accessed on Mar. 25, 2008, 1
`page.
`VMware, Inc., “WMware Workstation 5.0, Snapshots in a Process
`Tree.”
`<http://www.vmware.com/support/ws5/doc/ws_preserve_
`sshot_tree.html>. internet accessed on Mar. 25, 2008, | page.
`VMware, Inc., “VMware Workstation 5.5, What Files Make Up a
`Virtual
`Machine?” <http:/‘www.vmware.com/support/ws55/doc/
`ws_learning _files_in_a_vm.html>, internet accessed on Mar. 25,
`2008, 2 pages.
`Wikipedia, “Cluster (file system),” <http://en.wikipedia.org/wiki/
`Cluster_°28file_system%29>, internet accessed Jul. 25, 2008, 1
`page.
`Wikipedia, “Cylinder-head-sector,” <http://en.wikipedia.org/ wiki!
`
`Cylinder-head-sector>, internet accessed Jul. 22, 2008, 6 pages.
`Wikipedia, “File Allocation Table,” <http://en.wikipedia.org/wiki!
`File_ Allocation
`Table>, internet accessed on Jul. 25, 2008, 19
`pages.
`Wikipedia, “Logical Disk Manager,” <http://en.wikipedia.org/wiki/
`Logical_Disk_Manager>, internet accessed Mar. 26, 2008, 3 pages.
`Wikipedia, “Logical vol. Management,” <http://en.wikipedia.org/
`wiki/Logical_volume_management>, internet accessed on Mar. 26,
`2008, 5 pages.
`Wikipedia, “Storage Area Network,” <http://en.wikipedia.org/ wiki’
`
`Storage_area_network>, internet accessed on Oct. 24, 2008, 5
`pages.
`<http://en.wikipedia.org/wiki/
`“Virtualization,”
`Wikipedia,
`Virtualization>, internet accessed Mar. 18, 2008, 7 pages.
`
`* cited by examiner
`
`

`

`yuayed Sn
`
`ST0Z'SZ “Sny
`
`ZL JO T199US
`
`7d cc99IT6Sr1
`
`|
`
`|
`
`|
`|
`
`|
`
`_
`
`135a |
`|
`—140a |
`
`
`Virtual
`machine
`
`manager |
`
`一 115c
`
`7
`
`120c
`
`115d
`一 120d
`
`|
`|
`|
`|
`|
`|
`|
`|
`|
`圖 |
`
`[ae
`[sms
`
`+
`+
`
`
`
`108) virtual machine host 105a
`virtual machine 110a
`Lapp 小 盖
`Luss hy
`
`
`100
`
`| J
`
`|
`
`|
`|
`
`|
`
`115a
`
`120a
`
`一 115b
`
`一 120b
`
`virtual machine 110b
`
`
`
`
`production
`data store
`
`
`
`106
`
`
`101a
`
`，
`client
`
`
`
`pp 一 个 103a
`
`[os
`Ho
`
`
`
`
`101b
`
`
`client
`
`
`
`
`
`109a
`
`production
`data store
`
`102b
`physical machine B
`
`[app i -+103»
`[os
`prov
`
`
`
`
`production
`
`data store
`
`
`101c
`
`
`client
`
`
`
`
`
`人 109b
`
`
`
`
`
`
`eee
`
`102n
`
`hysical machine N
`
`101d
`re
`
`client
`[app 十 直 03n
`Les hy pt07 |
`
`
`
`
`101e
`
`
`
`
`
`
`client
`
`
`
`production
`data store
`
`m10Sn
`
`[
`
`|
`|
`|
`
`|
`
`|
`|
`|
`|
`
`FIG. 1A
`
`
`
`104A
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`virtual machine 110d
`
`Ls 小 一
`
`
`
`
`
`
`
`
`
`

`

`yuayed “SN
`
`SL0z ‘Sz “Sny
`
`ZL JO Z 9048
`
`7d cc99IT6Sr1
`
`106
`
`111
`
`
`un
`
`foment]
`module |
`
`
`
`|
`
`|
`
`|
`|
`|
`
`|
`|
`
`|
`|
`|
`
`|
`
`|
`
`se
`
`|
`
`FIG. 1B
`
`150
`
`/
`
`|
`
`|
`:
`
`—— — — 158
`160a
`
`cloud
`|
`| computing
`vendor A
`
`
`
`
`100
`
`
`cloud
`computing
`vendorB
`
`
`
`
`|
`|
`
`|
`
`eee
`
`
`
`ai
`
`
`
`| 一 computing
`|
`vendor M
`L_ 二 一
`一 一
`
`|
`
`|
`
`|
`|
`|
`
`|
`|
`|
`
`|
`|
`
`|
`
`|
`
`|
`
`|
`
`!
`
`|
`
`
`101a
`[wen
`
`
`
`101b
`
`
`
`
`
`
`
`client
`
`
`102a
`physical machine A
`
`十 十 103s
`L_app
`[L_ os 上 个 1o07a
`
`
`
`
`广 109a
`
`production
`data store
`
`102b
`
`physical machine B
`
`
`[app
`sw
`
`
`
`-oa
`pt07
`
`人
`
`
`
`Io
`1
`
`
`
`一
`
`
`|
`client
`
`
`
`
`
`
`
`|
`|
`
`
`
`
`
`
`|
`113
`migraticn
`module
`am | |
`
`
`
`|
`|
`
`|
`
`|
`
`production
`data store
`
`
`六 109b
`
`
`physical machine N
`
`103
`
`101d
`
`.
`app
`| 个
`103n
`
`client
`[es _ 上 om |
`
`
`
`
`
`|
`|
`
`
`101e
`
`|
`.
`
`|
`client
`production 一 正 109n
`|
`data store
`|
`
`a |
`
`
`
`
`
`

`

`U.S. Patent
`
`Aug. 25, 2015
`
`Sheet 3 of 12
`
`US 9,116,633 B2
`
`
`
`migration system
`
`200
`
`
`
`memory 214
`software 216
`
`
`
`components 218
`
`230
`
`physical configuration
`determination component
`
`
`
`
`215
`virtual machine creation
`component
`
`
`
`
`
`232
`
`virtual machine
`configuration component
`
`
`202
`
`
`CPU
`
`
`204
`
`
`computer-readable
`media drive
`
`206
`
`
`
`persistent storage
`device
`
`
`
`
`
`208
`
`|
`network connection
`device
`
`210
`
`
`
`
`254 management component
`
`
`
`
`
`
`
`
`
`
`234
`
`-
`-
`data migration
`component
`
`233
`
`IP management
`component
`
`
`
`
`DR test component
`
`
`
`257
`
`256
`
`
`
`API component
`
`
`252
`user interface component
`
`
`
`
`
`
`
`
`
`
`
`data 220
`221
`virtual machine creation
`data
`222
`configuration data
`
`
`224
`
`
`
`
`
`
`
`data migration data
`
`
`226
`IP management data
`
`
`
`
`
`227
`
`DR test data
`
`
`
`information input
`device
`
`
`
`212
`information output
`device
`
`FIG. 2
`
`

`

`U.S. Patent
`
`Aug. 25, 2015
`
`Sheet 4 of 12
`
`US 9,116,633 B2
`
` Ca
`
`
`
`300
`
`
`
`
`receive request to clone
`physical machine functionality
`on virtual machine
`
`305
`
`
`
`
`
`310
`
`
`create non-production copy of
`data from physical machine
`
`
`
`315
`
`
`
`automatically detect
`configuration of physical
`machine
`
`
`
`
`
`
`determine desired
`configuration and location of
`virtual machine
`
`
`
`
`
`325
`
`
`connect to cloud service
`provider, VM host, or VM
`manager and request
`provisioning of virtual machine
`with desired configuration
`
`
`
` install information management agent onto provisioned virtual
`
`
`
`320
`335
`
`adjust configuration of virtual
`machine to match desired
`configuration
`|
`
`
`
`
`
`340
`
`
`
`install application binaries on
`virtual machine
`
`
`
`345
`
`
`
`make data and metadata
`copied from physical machine
`available to virtual machine
`
`
`
`
`350
`update IP address mapping of
`physical machine and/or virtual
`machine
`
`
`
` C=)
`
`
`
`355
`
`
`track, protect, and apply
`existing information
`management policies to data
`on virtual machine
`
`
`
`
`
`360
`use virtual machine for DR
`testing
`
`
`
`
`
`
`
`330
`
`
`
`machine
`
`
`
`FIG. 3
`
`

`

`U.S. Patent
`
`Aug. 25, 2015
`
`Sheet 5 of 12
`
`US 9,116,633 B2
`
`
`create and use non-
`production copy of
`virtual machine
`
`405
`a7
`
`
`discover virtual machine
`
`400
`
`
`
` Y 410
`
`discover virtual disks and mapped
`LUNs associated with virtual
`machine
`
`YY
`412
`
`track relationship between virtual
`machine, virtual disks, and mapped
`LUNs
`
`
`
`
`
`
`
`
` Y 415
`
`take snapshots of virtual disks
`
`
`
`
`
`420
`
`take snapshots of mapped LUNs
`
`430
`425
`
`use snapshots to create one or
`snapshots preserved to permit in-
`more additional non-production
`place reversion
`copies of virtual machine
`
`
`
`
`
`
`
`
`
`
`
`
`435
`
`use snapshots or additional non-
`production copies to revert or
`restore virtual machine
`
`
`
`
`
` N Cou)
`
`
`
`

`

`U.S. Patent
`
`Aug. 25, 2015
`
`Sheet 6 of 12
`
`US 9,116,633 B2
`
`
`
`create one or more additional non-
`production copies of virtual machine
`
`
`
`500
`
`
`
`505
`mount snapshot(s) onto proxy
`virtual host
`v
`
`
`
`510
`
`
`
`bring up proxy virtual machine from
`mounted snapshots of virtual disks
`
`
`
`
`
` W
`
`515
`
`mount snapshot of mapped LUN
`and map LUN (e.g. via VRDM)
`
`
`
`
`
`
`
`
`
` M
`
`520
`
`
`
`read data out of proxy virtual
`machine using virtual machine host
`APls and create additional non-
`production copies of data/metadata
`
` Co)
`
`
`
`FIG. 5
`
`

`

`U.S. Patent
`
`Aug. 25, 2015
`
`Sheet 7 of 12
`
`US 9,116,633 B2
`
`600
`
`650
`
`
`
`use snapshot(s) to
`revert virtual machine
`
`
`
`
`
`605
`
`
`
`y
`
`revert virtual disks on host
`using previous snapshot
`各
`
`610
`
`
`
`revert mapped LUN to its original
`location using previous snapshot
`
`
`
`
`
`
`==
`
`
`
`FIG. 6A
`
`
`
`use other non-
`production copies to
`estore virtual machine
`
`
`
`
`
`‘
`
`655
`
`bring up virtual machine using non-
`production copies of virtual disks
`
`
`
`
`
`660
`restore an associated LUN using a
`non-production (e.g., backup) copy
`of the LUN
`
`
`
`
`
`665
`
`mount the restored LUN on the
`virtual machine (e.g., via a VRDM)
`
`
`
`
`
`
`C=)
`
`
`
`
`
`
`
`FIG. 6B
`
`

`

`U.S. Patent
`
`Aug. 25, 2015
`
`Sheet 8 of 12
`
`US 9,116,633 B2
`
`700
`
`708
`
`710
`
`712
`
`can
`
`716
`
`718
`
`
`Virtual
`,
`Mapped LUN
`Snap
`Other
`Machine ID
`| Mount Point | Virtual Disk ID
`PPS
`Copy Ref
`Copy Ref
`
`
`702 AL
`VM_1
`C:/
`VD 1
`VD_1_snap* | VD_1_bu*
`
`
`704A),
`VM_1
`D:/
`PL_1
`PL 1 snap* | PL_1_bu*
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`VM_2
`C7
`VD_2
`VD_2 snap* | VD 2 bu*
`
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Aug. 25, 2015
`
`Sheet 9 of 12
`
`US 9,116,633 B2
`
`800
`
`
`
`
`
`
`
`
`
`SERVICES
`
`/
`
` HOSTED
`
`
`
`
`INFORMATION MANAGEMENT
`SYSTEM
`
`
`
`
`870
`
`
`
`U.S. Patent
`
`Aug. 25, 2015
`
`Sheet 9 of 12
`
`US 9,116,633 B2
`
`800
`
`
`
`
`
`
`
`
`
`SERVICES
`
`/
`
` HOSTED
`
`
`
`
`INFORMATION MANAGEMENT
`SYSTEM
`
`
`
`
`870
`
`
`
`

`

`U.S. Patent
`
`Aug. 25, 2015
`
`Sheet 10 of 12
`
`US 9,116,633 B2
`
`
`
`905
`
`910
`
`
`Operating system
`
`
`
`
`Applicati
`pplication
`
`980
`Data
`management
`agent(s)
`
`
`
`
`
`\
`
` 915A-D
`
`i
`
`
`
`
`
`935
`
`
`Meta1
`
`Meta2
`
`Meta3
`
`SI
`
`
`
`.
`
`)
`
`
`
`
`
`
`Meta4
`
`“Meta5
`
`]
`
`ae
`
`
`
`
`
`2
`
`
`
`
`
`| PRODUCTION
`
`| NON-PRODUCTION
`INFORMATION MANAGEMENT
`SYSTEM
`
`
`
`
`
`
`
`955C 一 |
`
`930 一 |
`
`945C 一
`
`
`
`
`
`Information
`Management Data
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Aug. 25, 2015
`
`Sheet 11 of 12
`
`US 9,116,633 B2
`
`|
`
`|
`|
`
`1050A
`
`Global Storage Manager 1005
`
`
`
`
`
`
`
`
`
`1000
`
`1050C
`
`
`
`Information
`Management Cell
`
`
`
`
`
`Information
`Management Cell
`
`
`
`
`
`
`
`
`
`Information
`Management Cell
`
`
`
`1050D 1050E
`
`Information
`Management Cell
`
`
`
`
`
`
`
`
`
`Information
`Management Cell
`
`FIG. 10
`
`
`
`

`

`yuayed Sn
`
`ST0Z'SZ “Sny
`
`ZLTJoOZLl99dS
`
`z98899TT6 SA
`
`1118C
`
`|
`
`[tee
`
`1118B
`
`一 一 人
`| > |
`
`1118A
`
`855
`
`80
`
`865
`
`870
`
`
`
`HOSTED
`ERVICES
`
`
`
`
`
`本
`1
`1
`1122A
`
`918A
`
`918B
`
`918N
`
`1122B
`
`1122C
`
`FIG. 77
`
`
`
`
`
`
`storage manager 1102
`
`1145
`1150
`
`mgmt
`
`1050
`
`network || mgmt.
`
`
`
`1155
`1160
`
`
`data and
`policies
`
`AL
`
`~~.
`
`
`
`interface
`
`
`
`
`
`一 對 | jobs
`
`1
`1
`|
`|
`
`
`
`1116M
`、~、
`
`、、、
`
`~、~、
`
`~、
`
`人
`
`、、、1116B
`、
`1
`1116A
`、\、
`Media agent
`1110A
`Ps
`1110
`
`
`Ly
`Va
`
`
`
`VA
`,
`
`deduplication
`data
`media agent
`classification
`9
`database
`<
`
`
`
`1175
`1185
`1190 |
`/
`
`encryption
`
`compression ||
`
`distribution
`
`|
`
`|
`
`$
`外
`905A
`905B
`
`
`41105
`1105
`
`Distribution client
`Distribution client
`
`
`
`
`
`1 1
`1
`1
`1
`1
`1
`1
`1
`1
`1
`|
`
`1
`1 1
`1
`
`|
`
`1
`|
`1
`|
`1
`|
`!
`
`network
`Content
`media file
`indexing
`Systems
`
`1130
`1135
`40
`
`
`
`
`
`
`
`
`
`
`
`|
`
`980
`data mgmt.
`agent
`
`|
`
`
`
`VY
`90SN
`
`1105
`Distribution client
`
`
`
`980
`data mgmt.
`agent
`
`
`
`|
`
`980
`data mgmt.
`agent
`
`本
`*
`1
`1
`1
`1
`|
`1
`1
`1
`1
`L_--------- L__-------_--- L---------------- 1
`
`

`

`US 9,116,633 B2
`
`1
`INFORMATION MANAGEMENT OF
`VIRTUAL MACHINES HAVING MAPPED
`STORAGE DEVICES
`
`we
`
`CROSS-REFERENCE TO RELATED
`APPLICATION(S)
`
`The present application claims priority to and the benefit of
`US. Provisional Application No. 61/618,663 filed Mar. 30,
`2012, which is hereby incorporated herein by reference in its
`entirety.
`This application is related to assignee’s U.S. patent appli-
`cation Ser. No. 13/250,962, filed on Sep. 30, 2011, entitled
`“EFFICIENT DATA MANAGEMENT IMPROVEMENTS,
`SUCH AS DOCKING LIMITED-FEATURE DATA MAN-
`AGEMENT MODULES TO A FULL-FEATURED DATA
`MANAGEMENT SYSTEM”, and to U.S. Provisional Patent
`Application 61/618,579, entitled DATA STORAGE RECOV-
`ERY AUTOMATION, filed Mar. 30, 2012, both of which are
`hereby incorporated by reference herein in their entirety.
`
`BACKGROUND
`
`2
`A cloud service often provides an Application Program-
`ming Interface (“API”) or similar interface that enables a
`cloud service customer’s machines to access, modify, and
`otherwise manage the customer’s cloud services and data.
`Typically, cloud services are facilitated by virtualization. For
`example, in order to provide data storage, access, and com-
`putational resources for customers, a cloud service provider
`may operate a data center having multiple virtual machine
`hosts, each of which runs multiple virtual machines.
`When companies transition their existing in-house physi-
`cal IT infrastructure, such as existing application servers, to
`virtual machines and cloud services, they may experience
`numerous difficulties. For example, the company’s IT staff
`may be unfamiliar with virtualization software or cloud ser-
`vice APIs and therefore spend substantial internal resources
`familiarizing themselves with these. Alternatively, a compa-
`ny’s IT staff may be familiar with using only a single virtu-
`alization or cloud computing service vendor and therefore
`miss the opportunity to transition to a different, more cost-
`effective vendor. The internal IT staff may also incorrectly
`transition existing application servers or stored data to a new
`virtualized or cloud platform. Moreover, even if IT staff are
`familiar with numcrous virtualization and cloud service ven-
`dors, they may need to spend substantial time and energy
`cataloguing existing physical IT resources, designing and
`deploying comparable virtualization or cloud services plat-
`forms to replace those physical resources, porting existing
`applications to the new platforms, testing the new platforms,
`and redirecting client service requests to the new platforms.
`Some systems and software may convert a physical
`machine to a virtual machine. However, typically such sys-
`tems and software require that all of the production data of the
`physical machine be read and transmitted over a network,
`which requires substantial time and network bandwidth.
`These systems and software also typically require that the
`same data be stored twice: once at the source physical
`machine and once at the destination virtual machine. More-
`over, such systems and software typically must perform sub-
`stantial processing to convert the production data of the
`physical machine to a virtual disk file (e.g., a .vmdk file).
`Further, such systems and software typically cannot create a
`virtual machine from an earlier, point-in-time state of the
`physical machine, but can only create a virtual machine
`reflecting the state of the physical machine at the present time
`when the virtual machine is being created.
`The need exists for systems and methods that overcome the
`above problems, as well as systems and methods that provide
`additional benefits. Overall, the examples herein of some
`prior or related systems and methods and their associated
`limitations are intended to be illustrative and not exclusive.
`Other limitations of existing or prior systems and methods
`will become apparent to those of skill in the art upon reading
`the following Detailed Description.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`FIG. 1A is a block diagram illustrating an environment in
`which a system for migrating physical computing resources
`operates.
`FIG. 1B is a block diagram illustrating another environ-
`ment in which a system for migrating physical computing
`resources operates.
`FIG. 2 is a block diagram illustrating details of a migration
`system that can migrate physical computing resources to
`virtual machines and cloud service providers and provide
`disaster recovery testing functionality.
`
`Companies have the opportunity to reduce the complexity
`of their IT infrastructure and their IT costs by using virtual-
`ization technologies and cloud computing services. In gen-
`eral, virtualization refers to the simultaneous hosting of one
`or more operating systems on a physical computer. Such
`virtual
`operating
`systems and
`their
`associated
`virtual
`resources are called virtual machines. Virtualization software
`(or synonymously, a hypervisor or Virtual Machine Monitor),
`sits between the virtual machines and the hardware of the
`physical computer, which is called a virtual machine host.
`One example of virtualization software is ESX Server, by
`VMware, Inc. of Palo Alto, Calif. Other examples include
`Microsoft Virtual Server and Microsoft Windows Server
`Hyper-V, both by Microsoft Corporation of Redmond, Wash.,
`and Sun xVM by Oracle America Inc. of Santa Clara, Calif.
`Virtualization software provides to each virtual operating
`system virtual resources, such as virtual processors, virtual
`memory, virtual network devices, and virtual disks. Each
`virtual machine has one or more virtual disks to store the files,
`metadata, and other data used by the virtual machine. Virtu-
`alization software typically stores the data of virtual disks in
`files on the file system of the physical computer, called virtual
`machine disk files (in the case of VMware virtual servers) or
`virtual hard disk image files (in the case of Microsoft virtual
`servers). For example, VMware’s ESX Server provides the
`Virtual Machine File System (VMFS) for managing virtual
`machine disk files. A virtual machine reads data from and
`writes data to its virtual disk much the same way that an actual
`physical machine reads data from and writes data to an actual
`disk.
`Cloud computing services (or “cloud services”) provide 5
`computing as a service by sharing resources and software to
`computing devices over a network such as the Internet, typi-
`cally as a paid (e.g., metered or subscription) service. For
`example, cloud services provide storage resources, computa-
`tional power, software, and data access to their customers,
`who are generally unaware of the underlying structure of the
`physical devices and software that facilitate the services.
`Examples of cloud computing service vendors include Ama-
`zon Corporation of Seattle, Wash. (offering Amazon Web
`Services); Rackspace Corporation of San Antonio, Tex.; and
`Microsoft Corporation of Redmond, Wash. (offering Win-
`dows Azure).
`
`9
`
`40
`
`45
`
`:
`
`60
`
`

`

`US 9,116,633 B2
`
`we
`
`30
`
`3
`FIG. 3 is a flow diagram illustrating a process implemented
`by the migration system in connection with migrating physi-
`cal computing resources to one or more virtual machines.
`FIG. 4 is a flow diagram illustrating a process for creating
`and using a non-production copy of the data and metadata of
`a virtual machine having a mapped LUN.
`FIG. 5 is a flow diagram illustrating a process for creating
`one or more additional non-production copies of a virtual
`machine’s data and metadata from snapshots.
`FIG. 6A shows a process for using snapshots to revert a
`virtual machine having a mapped LUN to an earlier point in
`time.
`FIG. 6B shows a process for using non-production copies
`to restore a virtual machine that has one or more mapped
`LUNs.
`FIG. 7 illustrates an example of a virtual machine tracking
`data structure.
`FIG. 8
`is a block diagram illustrating an example of a
`suitable information management environment in which
`aspects of the inventive systems and methods may operate.
`FIG. 9 illustrates examples of production and non-produc-
`tion data that may be processed by the information manage-
`ment environment of F