indexing.py 15 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440
  1. """==============
  2. Array indexing
  3. ==============
  4. Array indexing refers to any use of the square brackets ([]) to index
  5. array values. There are many options to indexing, which give numpy
  6. indexing great power, but with power comes some complexity and the
  7. potential for confusion. This section is just an overview of the
  8. various options and issues related to indexing. Aside from single
  9. element indexing, the details on most of these options are to be
  10. found in related sections.
  11. Assignment vs referencing
  12. =========================
  13. Most of the following examples show the use of indexing when
  14. referencing data in an array. The examples work just as well
  15. when assigning to an array. See the section at the end for
  16. specific examples and explanations on how assignments work.
  17. Single element indexing
  18. =======================
  19. Single element indexing for a 1-D array is what one expects. It work
  20. exactly like that for other standard Python sequences. It is 0-based,
  21. and accepts negative indices for indexing from the end of the array. ::
  22. >>> x = np.arange(10)
  23. >>> x[2]
  24. 2
  25. >>> x[-2]
  26. 8
  27. Unlike lists and tuples, numpy arrays support multidimensional indexing
  28. for multidimensional arrays. That means that it is not necessary to
  29. separate each dimension's index into its own set of square brackets. ::
  30. >>> x.shape = (2,5) # now x is 2-dimensional
  31. >>> x[1,3]
  32. 8
  33. >>> x[1,-1]
  34. 9
  35. Note that if one indexes a multidimensional array with fewer indices
  36. than dimensions, one gets a subdimensional array. For example: ::
  37. >>> x[0]
  38. array([0, 1, 2, 3, 4])
  39. That is, each index specified selects the array corresponding to the
  40. rest of the dimensions selected. In the above example, choosing 0
  41. means that the remaining dimension of length 5 is being left unspecified,
  42. and that what is returned is an array of that dimensionality and size.
  43. It must be noted that the returned array is not a copy of the original,
  44. but points to the same values in memory as does the original array.
  45. In this case, the 1-D array at the first position (0) is returned.
  46. So using a single index on the returned array, results in a single
  47. element being returned. That is: ::
  48. >>> x[0][2]
  49. 2
  50. So note that ``x[0,2] = x[0][2]`` though the second case is more
  51. inefficient as a new temporary array is created after the first index
  52. that is subsequently indexed by 2.
  53. Note to those used to IDL or Fortran memory order as it relates to
  54. indexing. NumPy uses C-order indexing. That means that the last
  55. index usually represents the most rapidly changing memory location,
  56. unlike Fortran or IDL, where the first index represents the most
  57. rapidly changing location in memory. This difference represents a
  58. great potential for confusion.
  59. Other indexing options
  60. ======================
  61. It is possible to slice and stride arrays to extract arrays of the
  62. same number of dimensions, but of different sizes than the original.
  63. The slicing and striding works exactly the same way it does for lists
  64. and tuples except that they can be applied to multiple dimensions as
  65. well. A few examples illustrates best: ::
  66. >>> x = np.arange(10)
  67. >>> x[2:5]
  68. array([2, 3, 4])
  69. >>> x[:-7]
  70. array([0, 1, 2])
  71. >>> x[1:7:2]
  72. array([1, 3, 5])
  73. >>> y = np.arange(35).reshape(5,7)
  74. >>> y[1:5:2,::3]
  75. array([[ 7, 10, 13],
  76. [21, 24, 27]])
  77. Note that slices of arrays do not copy the internal array data but
  78. only produce new views of the original data.
  79. It is possible to index arrays with other arrays for the purposes of
  80. selecting lists of values out of arrays into new arrays. There are
  81. two different ways of accomplishing this. One uses one or more arrays
  82. of index values. The other involves giving a boolean array of the proper
  83. shape to indicate the values to be selected. Index arrays are a very
  84. powerful tool that allow one to avoid looping over individual elements in
  85. arrays and thus greatly improve performance.
  86. It is possible to use special features to effectively increase the
  87. number of dimensions in an array through indexing so the resulting
  88. array aquires the shape needed for use in an expression or with a
  89. specific function.
  90. Index arrays
  91. ============
  92. NumPy arrays may be indexed with other arrays (or any other sequence-
  93. like object that can be converted to an array, such as lists, with the
  94. exception of tuples; see the end of this document for why this is). The
  95. use of index arrays ranges from simple, straightforward cases to
  96. complex, hard-to-understand cases. For all cases of index arrays, what
  97. is returned is a copy of the original data, not a view as one gets for
  98. slices.
  99. Index arrays must be of integer type. Each value in the array indicates
  100. which value in the array to use in place of the index. To illustrate: ::
  101. >>> x = np.arange(10,1,-1)
  102. >>> x
  103. array([10, 9, 8, 7, 6, 5, 4, 3, 2])
  104. >>> x[np.array([3, 3, 1, 8])]
  105. array([7, 7, 9, 2])
  106. The index array consisting of the values 3, 3, 1 and 8 correspondingly
  107. create an array of length 4 (same as the index array) where each index
  108. is replaced by the value the index array has in the array being indexed.
  109. Negative values are permitted and work as they do with single indices
  110. or slices: ::
  111. >>> x[np.array([3,3,-3,8])]
  112. array([7, 7, 4, 2])
  113. It is an error to have index values out of bounds: ::
  114. >>> x[np.array([3, 3, 20, 8])]
  115. <type 'exceptions.IndexError'>: index 20 out of bounds 0<=index<9
  116. Generally speaking, what is returned when index arrays are used is
  117. an array with the same shape as the index array, but with the type
  118. and values of the array being indexed. As an example, we can use a
  119. multidimensional index array instead: ::
  120. >>> x[np.array([[1,1],[2,3]])]
  121. array([[9, 9],
  122. [8, 7]])
  123. Indexing Multi-dimensional arrays
  124. =================================
  125. Things become more complex when multidimensional arrays are indexed,
  126. particularly with multidimensional index arrays. These tend to be
  127. more unusual uses, but they are permitted, and they are useful for some
  128. problems. We'll start with the simplest multidimensional case (using
  129. the array y from the previous examples): ::
  130. >>> y[np.array([0,2,4]), np.array([0,1,2])]
  131. array([ 0, 15, 30])
  132. In this case, if the index arrays have a matching shape, and there is
  133. an index array for each dimension of the array being indexed, the
  134. resultant array has the same shape as the index arrays, and the values
  135. correspond to the index set for each position in the index arrays. In
  136. this example, the first index value is 0 for both index arrays, and
  137. thus the first value of the resultant array is y[0,0]. The next value
  138. is y[2,1], and the last is y[4,2].
  139. If the index arrays do not have the same shape, there is an attempt to
  140. broadcast them to the same shape. If they cannot be broadcast to the
  141. same shape, an exception is raised: ::
  142. >>> y[np.array([0,2,4]), np.array([0,1])]
  143. <type 'exceptions.ValueError'>: shape mismatch: objects cannot be
  144. broadcast to a single shape
  145. The broadcasting mechanism permits index arrays to be combined with
  146. scalars for other indices. The effect is that the scalar value is used
  147. for all the corresponding values of the index arrays: ::
  148. >>> y[np.array([0,2,4]), 1]
  149. array([ 1, 15, 29])
  150. Jumping to the next level of complexity, it is possible to only
  151. partially index an array with index arrays. It takes a bit of thought
  152. to understand what happens in such cases. For example if we just use
  153. one index array with y: ::
  154. >>> y[np.array([0,2,4])]
  155. array([[ 0, 1, 2, 3, 4, 5, 6],
  156. [14, 15, 16, 17, 18, 19, 20],
  157. [28, 29, 30, 31, 32, 33, 34]])
  158. What results is the construction of a new array where each value of
  159. the index array selects one row from the array being indexed and the
  160. resultant array has the resulting shape (number of index elements,
  161. size of row).
  162. An example of where this may be useful is for a color lookup table
  163. where we want to map the values of an image into RGB triples for
  164. display. The lookup table could have a shape (nlookup, 3). Indexing
  165. such an array with an image with shape (ny, nx) with dtype=np.uint8
  166. (or any integer type so long as values are with the bounds of the
  167. lookup table) will result in an array of shape (ny, nx, 3) where a
  168. triple of RGB values is associated with each pixel location.
  169. In general, the shape of the resultant array will be the concatenation
  170. of the shape of the index array (or the shape that all the index arrays
  171. were broadcast to) with the shape of any unused dimensions (those not
  172. indexed) in the array being indexed.
  173. Boolean or "mask" index arrays
  174. ==============================
  175. Boolean arrays used as indices are treated in a different manner
  176. entirely than index arrays. Boolean arrays must be of the same shape
  177. as the initial dimensions of the array being indexed. In the
  178. most straightforward case, the boolean array has the same shape: ::
  179. >>> b = y>20
  180. >>> y[b]
  181. array([21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34])
  182. Unlike in the case of integer index arrays, in the boolean case, the
  183. result is a 1-D array containing all the elements in the indexed array
  184. corresponding to all the true elements in the boolean array. The
  185. elements in the indexed array are always iterated and returned in
  186. :term:`row-major` (C-style) order. The result is also identical to
  187. ``y[np.nonzero(b)]``. As with index arrays, what is returned is a copy
  188. of the data, not a view as one gets with slices.
  189. The result will be multidimensional if y has more dimensions than b.
  190. For example: ::
  191. >>> b[:,5] # use a 1-D boolean whose first dim agrees with the first dim of y
  192. array([False, False, False, True, True])
  193. >>> y[b[:,5]]
  194. array([[21, 22, 23, 24, 25, 26, 27],
  195. [28, 29, 30, 31, 32, 33, 34]])
  196. Here the 4th and 5th rows are selected from the indexed array and
  197. combined to make a 2-D array.
  198. In general, when the boolean array has fewer dimensions than the array
  199. being indexed, this is equivalent to y[b, ...], which means
  200. y is indexed by b followed by as many : as are needed to fill
  201. out the rank of y.
  202. Thus the shape of the result is one dimension containing the number
  203. of True elements of the boolean array, followed by the remaining
  204. dimensions of the array being indexed.
  205. For example, using a 2-D boolean array of shape (2,3)
  206. with four True elements to select rows from a 3-D array of shape
  207. (2,3,5) results in a 2-D result of shape (4,5): ::
  208. >>> x = np.arange(30).reshape(2,3,5)
  209. >>> x
  210. array([[[ 0, 1, 2, 3, 4],
  211. [ 5, 6, 7, 8, 9],
  212. [10, 11, 12, 13, 14]],
  213. [[15, 16, 17, 18, 19],
  214. [20, 21, 22, 23, 24],
  215. [25, 26, 27, 28, 29]]])
  216. >>> b = np.array([[True, True, False], [False, True, True]])
  217. >>> x[b]
  218. array([[ 0, 1, 2, 3, 4],
  219. [ 5, 6, 7, 8, 9],
  220. [20, 21, 22, 23, 24],
  221. [25, 26, 27, 28, 29]])
  222. For further details, consult the numpy reference documentation on array indexing.
  223. Combining index arrays with slices
  224. ==================================
  225. Index arrays may be combined with slices. For example: ::
  226. >>> y[np.array([0,2,4]),1:3]
  227. array([[ 1, 2],
  228. [15, 16],
  229. [29, 30]])
  230. In effect, the slice is converted to an index array
  231. np.array([[1,2]]) (shape (1,2)) that is broadcast with the index array
  232. to produce a resultant array of shape (3,2).
  233. Likewise, slicing can be combined with broadcasted boolean indices: ::
  234. >>> y[b[:,5],1:3]
  235. array([[22, 23],
  236. [29, 30]])
  237. Structural indexing tools
  238. =========================
  239. To facilitate easy matching of array shapes with expressions and in
  240. assignments, the np.newaxis object can be used within array indices
  241. to add new dimensions with a size of 1. For example: ::
  242. >>> y.shape
  243. (5, 7)
  244. >>> y[:,np.newaxis,:].shape
  245. (5, 1, 7)
  246. Note that there are no new elements in the array, just that the
  247. dimensionality is increased. This can be handy to combine two
  248. arrays in a way that otherwise would require explicitly reshaping
  249. operations. For example: ::
  250. >>> x = np.arange(5)
  251. >>> x[:,np.newaxis] + x[np.newaxis,:]
  252. array([[0, 1, 2, 3, 4],
  253. [1, 2, 3, 4, 5],
  254. [2, 3, 4, 5, 6],
  255. [3, 4, 5, 6, 7],
  256. [4, 5, 6, 7, 8]])
  257. The ellipsis syntax maybe used to indicate selecting in full any
  258. remaining unspecified dimensions. For example: ::
  259. >>> z = np.arange(81).reshape(3,3,3,3)
  260. >>> z[1,...,2]
  261. array([[29, 32, 35],
  262. [38, 41, 44],
  263. [47, 50, 53]])
  264. This is equivalent to: ::
  265. >>> z[1,:,:,2]
  266. array([[29, 32, 35],
  267. [38, 41, 44],
  268. [47, 50, 53]])
  269. Assigning values to indexed arrays
  270. ==================================
  271. As mentioned, one can select a subset of an array to assign to using
  272. a single index, slices, and index and mask arrays. The value being
  273. assigned to the indexed array must be shape consistent (the same shape
  274. or broadcastable to the shape the index produces). For example, it is
  275. permitted to assign a constant to a slice: ::
  276. >>> x = np.arange(10)
  277. >>> x[2:7] = 1
  278. or an array of the right size: ::
  279. >>> x[2:7] = np.arange(5)
  280. Note that assignments may result in changes if assigning
  281. higher types to lower types (like floats to ints) or even
  282. exceptions (assigning complex to floats or ints): ::
  283. >>> x[1] = 1.2
  284. >>> x[1]
  285. 1
  286. >>> x[1] = 1.2j
  287. <type 'exceptions.TypeError'>: can't convert complex to long; use
  288. long(abs(z))
  289. Unlike some of the references (such as array and mask indices)
  290. assignments are always made to the original data in the array
  291. (indeed, nothing else would make sense!). Note though, that some
  292. actions may not work as one may naively expect. This particular
  293. example is often surprising to people: ::
  294. >>> x = np.arange(0, 50, 10)
  295. >>> x
  296. array([ 0, 10, 20, 30, 40])
  297. >>> x[np.array([1, 1, 3, 1])] += 1
  298. >>> x
  299. array([ 0, 11, 20, 31, 40])
  300. Where people expect that the 1st location will be incremented by 3.
  301. In fact, it will only be incremented by 1. The reason is because
  302. a new array is extracted from the original (as a temporary) containing
  303. the values at 1, 1, 3, 1, then the value 1 is added to the temporary,
  304. and then the temporary is assigned back to the original array. Thus
  305. the value of the array at x[1]+1 is assigned to x[1] three times,
  306. rather than being incremented 3 times.
  307. Dealing with variable numbers of indices within programs
  308. ========================================================
  309. The index syntax is very powerful but limiting when dealing with
  310. a variable number of indices. For example, if you want to write
  311. a function that can handle arguments with various numbers of
  312. dimensions without having to write special case code for each
  313. number of possible dimensions, how can that be done? If one
  314. supplies to the index a tuple, the tuple will be interpreted
  315. as a list of indices. For example (using the previous definition
  316. for the array z): ::
  317. >>> indices = (1,1,1,1)
  318. >>> z[indices]
  319. 40
  320. So one can use code to construct tuples of any number of indices
  321. and then use these within an index.
  322. Slices can be specified within programs by using the slice() function
  323. in Python. For example: ::
  324. >>> indices = (1,1,1,slice(0,2)) # same as [1,1,1,0:2]
  325. >>> z[indices]
  326. array([39, 40])
  327. Likewise, ellipsis can be specified by code by using the Ellipsis
  328. object: ::
  329. >>> indices = (1, Ellipsis, 1) # same as [1,...,1]
  330. >>> z[indices]
  331. array([[28, 31, 34],
  332. [37, 40, 43],
  333. [46, 49, 52]])
  334. For this reason it is possible to use the output from the np.nonzero()
  335. function directly as an index since it always returns a tuple of index
  336. arrays.
  337. Because the special treatment of tuples, they are not automatically
  338. converted to an array as a list would be. As an example: ::
  339. >>> z[[1,1,1,1]] # produces a large array
  340. array([[[[27, 28, 29],
  341. [30, 31, 32], ...
  342. >>> z[(1,1,1,1)] # returns a single value
  343. 40
  344. """
  345. from __future__ import division, absolute_import, print_function