58

I have a game state in Python with about 1000 objects (planetary systems + stars + planets), and I need to copy it and apply a bunch of transformations to it when requested. However, at about 1 request/second, this is taking up 24.63% of my runtime. How can I make it go fast? Note that copying less is not an option, since the transforms touch just about everything.

EDIT: got it down to 8% with judicious implementation of __deepcopy__ on things. Still, not good enough. (Good enough is 1% or less, I plan on throwing many more things at this.) timeit says 41.8ms per deepcopy().

Improve this question
10
  • Alternatively to copying the state, you could create an action queue: Use the current state to determine the next actions and effects for all the objects without applying them right away, then apply all those actions in one batch, then calculate the actions for the next 'turn', etc.
    – tobias_k
    Commented Jul 15, 2014 at 11:21
  • @tobias_k That's actually exactly what my pile of transforms is doing, but I need to do it afresh quite often, hence the copying.
    – Electro
    Commented Jul 15, 2014 at 11:22
  • Then I don't understand the question: Are you applying different sets of actions to the same state. maybe to see which ones come out best?
    – tobias_k
    Commented Jul 15, 2014 at 11:23
  • Yup. Well, not to see which ones come out best, but that is what I am doing.
    – Electro
    Commented Jul 15, 2014 at 11:24
  • Another idea: Maybe you can create some sort of "diff-state", i.e. like "this is x and this is y and everything else is the same as in parent state z".
    – tobias_k
    Commented Jul 15, 2014 at 11:27

5 Answers 5

Reset to default
74

Actually, deepcopy is very slow. But we can use json, ujson, or cPickle. we can use json/cPickle to dump an object, and load it later. This is my test:

Total time: 3.46068 s
File: test_deepcopy.py
Function: test at line 15
Line #   Hits          Time Per Hit   % Time  Line Contents
==============================================================
15                                             @profile
16                                             def test():
17       100       957585   9575.9     27.7        b = deepcopy(a)
18       100          862      8.6      0.0        c = copy(a)
19       100        42295    422.9      1.2        d = ujson.loads(ujson.dumps(a))
20       100        85040    850.4      2.5        e = json.loads(json.dumps(a))
21       100      2323465  23234.7     67.1        f = pickle.loads(pickle.dumps(a, -1))
22       100        51434    514.3      1.5        g = cPickle.loads(cPickle.dumps(a, -1))

as what we can see, json/ujson/cPickle is faster than deepcopy, but pickle...

Improve this answer
6
  • 1
    That is scary. What does deepcopy do that cPickle doesn't?
    – Electro
    Commented Apr 9, 2015 at 11:05
  • 2
    I don't know... you can find something useful here :)
    – cherish
    Commented Apr 13, 2015 at 1:37
  • 6
    I did some testing and found that deepcopy outperformed json on lists of dictionaries while json outperformed deepcopy on large nested dictionaries.
    – wrkyle
    Commented Mar 16, 2018 at 0:11
  • 14
    Please note that for python3, pickle is what's listed as cPickle here.
    – Karl Bartel
    Commented Jan 6, 2020 at 16:46
  • 5
    Be careful with json.loads(json.dumps(a)) (probably for ujson as well), they will always consider keys as strings. If a key is not a string, it'll convert to a string.
    – Rikard Olsson
    Commented Jan 20, 2022 at 13:35
10

If you create your own class to hold these objects you can create your own methods that work with copy and deep copy (see https://github.com/RafeKettler/magicmethods):

class MyClass():
    def __copy__(self):
        copy_object = MyClass()
        return copy_object

    def __deepcopy__(self, memodict={}):
        copy_object = MyClass()
        copy_object.value = self.value
        return copy_object

if __name__ == "__main__":
    my_inst = MyClass()
    print(copy.deepcopy(my_inst))

Here is a similar description from the previous broken link.

Copying

Sometimes, particularly when dealing with mutable objects, you want to be able to copy an object and make changes without affecting what you copied from. This is where Python's copy comes into play. However (fortunately), Python modules are not sentient, so we don't have to worry about a Linux-based robot uprising, but we do have to tell Python how to efficiently copy things.

__copy__(self)

Defines behavior for copy.copy() for instances of your class. copy.copy() returns a shallow copy of your object -- this means that, while the instance itself is a new instance, all of its data is referenced -- i.e., the object itself is copied, but its data is still referenced (and hence changes to data in a shallow copy may cause changes in the original).

__deepcopy__(self, memodict={})

Defines behavior for copy.deepcopy() for instances of your class. copy.deepcopy() returns a deep copy of your object -- the object and its data are both copied. memodict is a cache of previously copied objects -- this optimizes copying and prevents infinite recursion when copying recursive data structures. When you want to deep copy an individual attribute, call copy.deepcopy() on that attribute with memodict as the first argument. What are some use cases for these magic methods? As always, in any case where you need more fine-grained control than what the default behavior gives you. For instance, if you are attempting to copy an object that stores a cache as a dictionary (which might be large), it might not make sense to copy the cache as well -- if the cache can be shared in memory between instances, then it should be.

Improve this answer
2
  • That is actually what I was just pursuing. However, I only got it down to 8% of my runtime. Things learned: deepcopying a sortedcontainers.SortedListWithKey is slow, listify it first. Also copying itertools.count() is slow, which probably applies to generators in general.
    – Electro
    Commented Jul 15, 2014 at 12:07
  • I found a similar description for copying and added it to the bottom of my answer. I also found that there was a github repository for that link.
    – justengel
    Commented Jan 16, 2017 at 14:02
5

The built-in module marshal is faster than ujson and supports more native types like set, tuple, and non-string dict keys. It's not portable across Python versions (similarly to pickle), but for locally cloning data that's not an issue.

To show this, I took @BPL's test program from another answer and added marshal to the compared cases, and ran it on an ARMv6-compatible processor.

Added case:

print(timeit.timeit('marshal.loads(marshal.dumps(dataset))',
       setup='from __main__ import marshal, dataset', number=1))

Results (Marshal is the fastest for all cases):

2.7.14 (default, Mar  6 2019, 13:27:55)
[GCC 7.3.0]
Comparing deepcopy/marshal/ujson/json using random dataset(0,0,1), length 2
0.000588178634644
0.000134944915771
0.000258922576904
0.00113606452942
()
Comparing deepcopy/marshal/ujson/json using random dataset(0,0,3), length 2
0.000546932220459
0.000134944915771
0.000180006027222
0.00120401382446
()
Comparing deepcopy/marshal/ujson/json using random dataset(0,0,5), length 2
0.000545978546143
0.000128984451294
0.000185966491699
0.00106000900269
()
Comparing deepcopy/marshal/ujson/json using random dataset(0,2,1), length 50
0.00154900550842
0.000281810760498
0.000414848327637
0.00174903869629
()
Comparing deepcopy/marshal/ujson/json using random dataset(0,2,3), length 242
0.00655102729797
0.000789880752563
0.00133085250854
0.00432300567627
()
Comparing deepcopy/marshal/ujson/json using random dataset(0,2,5), length 1010
0.0514280796051
0.0015549659729
0.00413513183594
0.0148711204529
()
Comparing deepcopy/marshal/ujson/json using random dataset(0,4,1), length 172
0.00250005722046
0.000365018844604
0.000761985778809
0.00263404846191
()
Comparing deepcopy/marshal/ujson/json using random dataset(0,4,3), length 2892
0.0329101085663
0.00363397598267
0.0110101699829
0.0262169837952
()
Comparing deepcopy/marshal/ujson/json using random dataset(0,4,5), length 46412
0.616458892822
0.0826110839844
0.189103841782
0.504135131836
()
Comparing deepcopy/marshal/ujson/json using random dataset(2,0,1), length 2
0.000693082809448
0.000132083892822
0.000182867050171
0.00107002258301
()
Comparing deepcopy/marshal/ujson/json using random dataset(2,0,3), length 2
0.000566005706787
0.000132083892822
0.000180959701538
0.00107598304749
()
Comparing deepcopy/marshal/ujson/json using random dataset(2,0,5), length 2
0.000562906265259
0.000128984451294
0.000184059143066
0.00118517875671
()
Comparing deepcopy/marshal/ujson/json using random dataset(2,2,1), length 258
0.00405406951904
0.000534057617188
0.00124287605286
0.00309610366821
()
Comparing deepcopy/marshal/ujson/json using random dataset(2,2,3), length 1058
0.026270866394
0.00180387496948
0.00363302230835
0.0096640586853
()
Comparing deepcopy/marshal/ujson/json using random dataset(2,2,5), length 4338
0.0778729915619
0.00682806968689
0.0151469707489
0.0468928813934
()
Comparing deepcopy/marshal/ujson/json using random dataset(2,4,1), length 716
0.00720596313477
0.00100684165955
0.0215280056
0.0062358379364
()
Comparing deepcopy/marshal/ujson/json using random dataset(2,4,3), length 11468
0.112984895706
0.0238728523254
0.0448131561279
0.0874760150909
()
Comparing deepcopy/marshal/ujson/json using random dataset(2,4,5), length 183628
1.83552503586
0.407335042953
0.617804050446
1.65498495102
()
Comparing deepcopy/marshal/ujson/json using random dataset(4,0,1), length 2
0.000571012496948
0.000132083892822
0.000189781188965
0.00121593475342
()
Comparing deepcopy/marshal/ujson/json using random dataset(4,0,3), length 2
0.000757932662964
0.000131130218506
0.000180959701538
0.00144195556641
()
Comparing deepcopy/marshal/ujson/json using random dataset(4,0,5), length 2
0.00056791305542
0.000132083892822
0.000184059143066
0.00107407569885
()
Comparing deepcopy/marshal/ujson/json using random dataset(4,2,1), length 430
0.00451302528381
0.00053596496582
0.00142502784729
0.00343203544617
()
Comparing deepcopy/marshal/ujson/json using random dataset(4,2,3), length 1730
0.0259549617767
0.00232696533203
0.00387692451477
0.0187470912933
()
Comparing deepcopy/marshal/ujson/json using random dataset(4,2,5), length 7026
0.112207174301
0.0119769573212
0.0211799144745
0.0547370910645
()
Comparing deepcopy/marshal/ujson/json using random dataset(4,4,1), length 1684
0.00609397888184
0.00121903419495
0.00452899932861
0.00959086418152
()
Comparing deepcopy/marshal/ujson/json using random dataset(4,4,3), length 26828
0.19367814064
0.0293428897858
0.0688338279724
0.140627145767
()
Comparing deepcopy/marshal/ujson/json using random dataset(4,4,5), length 433484
3.54843020439
0.590909004211
1.09412097931
2.72070598602
Improve this answer
2
  • 1
    Thanks. marshal is the fastest of all the mentioned alternatives in this thread, slightly faster than cPickle as well.
    – Marius
    Commented Feb 12, 2022 at 21:56
  • Worth noting that marshal doesn't support cyclical objects that refer to themselves (pickle does I believe), but json serialization won't either.
    – Will S
    Commented Aug 7, 2023 at 13:29
4

I've made a fast experiment comparing both deepcopy/json/ujson for several cases and my results contradicts @cherish's ones on certain cases, posting the little experiment here:

import ujson
import timeit
import json
import random
import string
import copy
import ujson
import sys


def random_string(N):
    return ''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(N))


def random_json(width=5, height=5, levels=1):
    dct = {}
    lst = [random_string(4) for i in range(width)]
    lst2 = [random.randint(0, 10000) for i in range(width)]
    lst3 = [bool(random.randint(0, 1)) for i in range(width)]
    for j in range(height):
        dct[str(j)] = lst
        dct[str(width+j)] = lst2
        dct[str(2*width+j)] = lst3

    for i in range(levels):
        new_dct = {}
        for j in range(height):
            new_dct[str(j)] = dct
        dct = json.loads(json.dumps(new_dct))

    return new_dct

if __name__ == "__main__":
    print(sys.version)
    levels = 3
    for i in range(15):
        dataset = random_json(i, i, levels)
        print("Comparing deepcopy/ujson/json using random dataset({},{},{}), length {}".format(i,i,levels, len(json.dumps(dataset))))
        print(timeit.timeit('copy.deepcopy(dataset)',
                            setup='from __main__ import copy, dataset', number=10))
        print(timeit.timeit('ujson.loads(ujson.dumps(dataset))',
                            setup='from __main__ import ujson, dataset', number=10))
        print(timeit.timeit('json.loads(json.dumps(dataset))',
                            setup='from __main__ import json, dataset', number=10))
        print()

And the results would be:

3.6.4 (v3.6.4:d48eceb, Dec 19 2017, 06:04:45) [MSC v.1900 32 bit (Intel)]
Comparing deepcopy/ujson/json using random dataset(0,0,3), length 2
2.6842977659931844e-05
0.00012039864979822371
7.776568527950847e-05

Comparing deepcopy/ujson/json using random dataset(1,1,3), length 63
0.0002731667726569534
3.552747043226263e-05
0.00012987264191349377

Comparing deepcopy/ujson/json using random dataset(2,2,3), length 1106
0.0011858280130946362
0.00034974820892205325
0.0007093651596308467

Comparing deepcopy/ujson/json using random dataset(3,3,3), length 6834
0.0042218477363672215
0.0021178319874343293
0.003378267688436718

Comparing deepcopy/ujson/json using random dataset(4,4,3), length 26572
0.011379054029782284
0.006288757016181971
0.009920059244030693

Comparing deepcopy/ujson/json using random dataset(5,5,3), length 79210
0.028879491215043435
0.027906433274870912
0.029595961868760734

Comparing deepcopy/ujson/json using random dataset(6,6,3), length 183678
0.047142979515255284
0.04682125853300759
0.06791747047568517

Comparing deepcopy/ujson/json using random dataset(7,7,3), length 395528
0.08239215142913198
0.09871347134571351
0.15347433002098887

Comparing deepcopy/ujson/json using random dataset(8,8,3), length 764920
0.1351954464835896
0.19448842613700734
0.3020533693660834

Comparing deepcopy/ujson/json using random dataset(9,9,3), length 1356570
0.24560258734724671
0.44074906118659407
0.5705849913806413

Comparing deepcopy/ujson/json using random dataset(10,10,3), length 2287770
0.3237815755327835
0.61104051671153
0.8698565598118777

Comparing deepcopy/ujson/json using random dataset(11,11,3), length 3598750
0.4958284828467452
0.9472223636741877
1.2514314609961668

Comparing deepcopy/ujson/json using random dataset(12,12,3), length 5636414
0.6261448233909714
1.4066722957969802
1.8636325417418167

Comparing deepcopy/ujson/json using random dataset(13,13,3), length 8220800
0.8396582099444547
2.061675688670409
2.755659427352441

Comparing deepcopy/ujson/json using random dataset(14,14,3), length 12018290
1.0951926990258762
2.96703050743886
4.088875914783021

Conclusion from this little experiment is:

  • When dictionaries are small ones time(ujson)<time(json)<time(deepcopy)
  • When dictionaries are big ones time(deepcopy)<time(ujson)<time(json)

So it depends the number of copies you're making per second and which type of dictionary you're dealing with, you'll prefer switching between deepcopy or ujson.

Improve this answer
1

You can provide your own copy functions to the objects such that you won't need deep copy. deep copy inspects every object to check what needs to be copied. This is an expensive operation.

Improve this answer
4
  • I do need a deep copy of everything. The bits which are transformed are also the most numerous ones, and are hence causing deepcopy() to do more work.
    – Electro
    Commented Jul 15, 2014 at 11:15
  • @Electro I think what he meant is to write your own deep copy method that does copy everything, just without the reclection/introspection part.
    – tobias_k
    Commented Jul 15, 2014 at 11:25
  • I fail reading comprehension. Oops. Hmm, but is the introspection really the most expensive operation of deepcopy()?
    – Electro
    Commented Jul 15, 2014 at 11:28
  • 2
    Yes, deepcopy() has to make sure no reference loops will appear etc... There is also a lot of bookkeeping involved. See here, here and here.
    – Bort
    Commented Jul 15, 2014 at 11:35

Not the answer you're looking for? Browse other questions tagged or ask your own question.