You must have already seen and worked with some of them. In data science, you will often need to change the type of your data so that it becomes easier to use and work with. The type defines the operations that can be done on the data and the structure in which you want the data to be stored. Data types are a classification of data that tells the compiler or the interpreter how you want to use the data. Return bits a # this matmult is the key line of codeīits = d.iloc # read bits from my pandas dataframeĭ = np.Every value in Python has a data type. M,n = bits.shape # number of columns is needed, not bits.sizeĪ = 2**np.arange(n) # -1 reverses array of powers of 2 of same length as bits The new - quite simple - code runs 100K rows x 18 binary columns in ~1 sec ET on my host which is "mission accomplished" for me: '''įast way is vectorized matmult. So it can use a lot of processors and cores, if you have it. Secondly, matmult often runs in parallel without you seeing it, on many-core processors depending on your host configuration, especially when OpenBLAS or other BLAS is present for numpy to use on matrix algebra like this matmult. That's the real change in the python code.
The new code is vectorized, first of all. Well, next week I need to upgrade from 100K rows to 20M rows, so ~10 hours of running time was not going to be fast enough for me. The original code that runs one-row-at-a-time took ~3 minutes to run 100K rows of 18 columns in my pandas dataframe. I extended the good dot product solution of to run ~180x faster on my host, by using vectorized matrix multiplication code.
0 kommentar(er)
