Window size is too large while using ostinato

[ChrisCalvin]

I need help with handling a ValueError thrown from ostinato. I am passing a list of time series that were grouped using stumpy.mass and I am trying to find the most conserved part of those similar time series using ostinato. The issue I am having is with the window size being passed into ostinato. After some successful outcomes, the window size exceeds the max_size and throws the exception. My guess is that the conserved pattern is found near the end of the sub_seq and cannot complete the seq of length m before running out of terms. Any help is appreciated.

My output looks like this:

NN_Consensus_1
0 (0.04452651483159271, 24, 13)
Name: NN_Consensus_1, dtype: object
NN_Consensus_2
0 (0.026033168725705002, 2, 11)
Name: NN_Consensus_2, dtype: object
NN_Consensus_3
0 (0.038464504826220734, 2, 1)
Name: NN_Consensus_3, dtype: object
NN_Consensus_4
0 (0.02434462169698897, 11, 4)
Name: NN_Consensus_4, dtype: object
NN_Consensus_5
0 (0.04468526784183163, 21, 0)
Name: NN_Consensus_5, dtype: object
NN_Consensus_6
0 (0.02081740687251015, 7, 13)
Name: NN_Consensus_6, dtype: object
NN_Consensus_7
0 (0.036600762648718116, 7, 0)
Name: NN_Consensus_7, dtype: object
NN_Consensus_8
Reducing the size of m.

---

ValueError Traceback (most recent call last)
/tmp/ipykernel_7316/3817041441.py in get_neighbor_consensus(NN_list, df_motif, m)
53 try:
---> 54 radius, Ts_idx, subseq_idx = stumpy.ostinato(Ts_df.values, m)
55 df_consensus[name] = [(radius, Ts_idx, subseq_idx)]

~/anaconda3/envs/tf_gpu/lib/python3.9/site-packages/stumpy/core.py in inner_wrapper(*args, **kwargs)
126 if not is_same_params or kwargs.get("normalize", True):
--> 127 return norm(*args, **kwargs)
128 else:

~/anaconda3/envs/tf_gpu/lib/python3.9/site-packages/stumpy/ostinato.py in ostinato(Ts, m, normalize)
306 for i, T in enumerate(Ts):
--> 307 Ts[i], M_Ts[i], Σ_Ts[i] = core.preprocess(T, m)
308

~/anaconda3/envs/tf_gpu/lib/python3.9/site-packages/stumpy/core.py in preprocess(T, m)
1440 check_dtype(T)
-> 1441 check_window_size(m, max_size=T.shape[-1])
1442

~/anaconda3/envs/tf_gpu/lib/python3.9/site-packages/stumpy/core.py in check_window_size(m, max_size)
443 if max_size is not None and m > max_size:
--> 444 raise ValueError(f"The window size must be less than or equal to {max_size}")
445

ValueError: The window size must be less than or equal to 6

During handling of the above exception, another exception occurred:

ValueError Traceback (most recent call last)
/tmp/ipykernel_7316/3817041441.py in
68 df_motif = pd.read_csv(filepath+filename, index_col=[0], header=[0])
69 NN_list = get_neighbors(df_motif)
---> 70 cluster = get_neighbor_consensus(NN_list, df_motif, 10)
71 save_filename = peak+'_'+dim+'_NN_Consensus.csv'
72 cluster.to_csv(filepath+save_filename)

/tmp/ipykernel_7316/3817041441.py in get_neighbor_consensus(NN_list, df_motif, m)
56 except ValueError:
57 print('Reducing the size of m.')
---> 58 radius, Ts_idx, subseq_idx = stumpy.ostinato(Ts_df.values, m-1)
59 df_consensus[name] = [(radius, Ts_idx, subseq_idx)]
60 print(df_consensus[name])

~/anaconda3/envs/tf_gpu/lib/python3.9/site-packages/stumpy/core.py in inner_wrapper(*args, **kwargs)
125 is_same_params = _compare_parameters(norm, non_norm, exclude=exclude)
126 if not is_same_params or kwargs.get("normalize", True):
--> 127 return norm(*args, **kwargs)
128 else:
129 kwargs = {k: v for k, v in kwargs.items() if k != "normalize"}

~/anaconda3/envs/tf_gpu/lib/python3.9/site-packages/stumpy/ostinato.py in ostinato(Ts, m, normalize)
305 Σ_Ts = [None] * len(Ts)
306 for i, T in enumerate(Ts):
--> 307 Ts[i], M_Ts[i], Σ_Ts[i] = core.preprocess(T, m)
308
309 bsf_radius, bsf_Ts_idx, bsf_subseq_idx = _ostinato(Ts, m, M_Ts, Σ_Ts)

~/anaconda3/envs/tf_gpu/lib/python3.9/site-packages/stumpy/core.py in preprocess(T, m)
1439 T = np.asarray(T)
1440 check_dtype(T)
-> 1441 check_window_size(m, max_size=T.shape[-1])
1442
1443 T[np.isinf(T)] = np.nan

~/anaconda3/envs/tf_gpu/lib/python3.9/site-packages/stumpy/core.py in check_window_size(m, max_size)
442
443 if max_size is not None and m > max_size:
--> 444 raise ValueError(f"The window size must be less than or equal to {max_size}")
445
446

ValueError: The window size must be less than or equal to 6

​

[seanlaw]

@ChrisCalvin Welcome to the STUMPY community. It's hard to know what is causing this issue without a minimum-reproducible-example (with data) but, based on what the error is saying:

ValueError: The window size must be less than or equal to 6

This implies that your window size, m, is too large and, in fact, is larger than the length of one (or more) of the time series in your list of time series. Note that Ts_df must be a Python list of time series. Based on your usage of Ts_df.values, this implies that Ts_df is NOT a Python list and is, instead, a Pandas dataframe with multiple time series in each column. This will not work with ostinato and you will get unexpected (bad) results. Instead of passing Ts_df into stumpy.ostinato, you will need to convert each column (series) of your dataframe into a 1-dimensional numpy array:

list_of_arrays = [Ts_df[col].values for col in Ts_df]
radius, Ts_idx, subseq_idx = stumpy.ostinato(list_of_arrays, m)

Again, without more information, it will be hard to debug. Additionally, I noticed that you used for list in NN_list but note that you are replacing Python's builtin list function with some integer and this is bad. You should avoid using the word list as a variable. I hope that helps.

[ChrisCalvin]

Thank you Sean!! Changing the df to a list fixed the problem.