pyCoda

Submodules

pyCoda.cross_correlation module

pyCoda.data module

Implements the global import of all data

Created on Mon Dec 26 20:51:08 2016 @author: rwilson

class pyCoda.data.utilities

Bases: object

Collection of functions intended for data related processes.

DB_group_names(Database, group_name=None)

Read in group names found within group. If group is not provided the upper folder structure will be read from.

Parameters:

Database : str

Relative location of database

group_name : str

The expected attribute name/s

Returns:

group_names : list

Group names found within the group

Notes

Add some additional error checks

DB_attrs_save(Database, dictionary)

Save attribute to database head.

Parameters:

Database : str

Relative location of database

dictionary : dict

Dictionary of attributes

Notes

Add some additional error checks

DB_attrs_load(Database, attrs_names)

Read attribute from database head.

Parameters:

Database : str

Relative location of database

attrs_names : list(str)

The expected attribute name/s

Returns:

dict_attri : dict

The returned dictionary of attribute/s from the database

Notes

Add some additional error checks

DB_pd_data_load(Database, group, cols=None, whereList=None)

Loads in a pandas dataframe stored in group from the Database.

Parameters:

Database : str

Relative location of database

group : str

The expected group name

cols : list(str) / list(int)

If not None, will limit the return columns, only applicable for table format database. For fixed format database only int accepted

whereList : list of Term (or convertable) objects or slice(from, to)

The conditional import of data, example [‘index>11’, ‘index<20’], only applicable for table format database. For fixed format database only a slice object is applicable and will use the row index numbers not the index values (i.e. df.iloc vs df.loc)

Returns:

group_df : DataFrame

The PV data stored in the group PVdata as a pandas dataframe

TSsurvey = pd.read_hdf(h5file, ‘survey20180312093545’,

columns=[(1,1), (1,2)], # Load specific columns where = [‘index>11’, ‘index<20’]) # Load index 11 -> 20

DB_pd_data_save(Database, group, df)

Saves in a pandas dataframe stored in group from the Database.

Parameters:

Database : str

Relative location of database

group : str

The expected group name

df : DateFrame

Pandas DataFrame to be stored in h5 file Database

PV_TS_DB_merge(CC, PVdata, mergeOnIndex=False)

Merge/concatenate based on the time axis. The expected structures of ‘PVdata’ and ‘CC’ DataFrames is a Time Stamp axis and a Time index level on which the concatenation takes place.

Parameters:

CC : DataFrame of list(DataFrame)

Expected to contain the processed data or a list of DataFrame with processed data. The index must be a timestam.

PVdata : DataFrame

A single DataFrame containing the corresponding perturbation information and must contain a column Time Stamp, which will be used during the concatentation/merge.

mergeOnIndex : Default False

Merge based on the index values. Currently only works when a single CC DataFrame is provided, and not for a list of DF.

Returns:

PV_CC_df : DataFrame

Merged/concatenated dataframe of both PV and Coda processed data.

column_values : array

of tuples defining the multi-level indecies of the CC data.

DB_COL_stats(DF, colList, baseName, stats=['mean', 'std'], norm=False)

Extract stats from multiple columns with a CommonKey.

Parameters:

DF : DataFrame

Dataframe from which statistics will be generated.

colList : str

a list of columns from which the stats will be made.

baseName : str

The base name of the new columns to which ‘_[stats]’ will be appended.

stats : list

A list of strings containing the requested stats to be generated for columns with CommonKey.

norm : list

Perform a min-max norm of the added statistic between 0 and 1.

Returns:

DF : DataFrame

Original dataframe plus columns containing requested stats.

CC_lag(CC_df, period, units='us', relVel=True)

Convert CC lag or First Break Picked data from number of sample points to time or relative velocity change based on:

\[\dfrac{\delta v}{v} = \dfrac{\delta t}{t}\]

Expected column names should either contain ‘lag’ in the last of a tuple, eg. col[-1] in the case of lag correction, or ‘FBP’ in the case of First Break Picking correction. If a ‘FBP’ correction is required, then the correct input initial velocity should be given.

Parameters:

CC_df : DataFrame

Dataframe from which statistics will be generated. A three level dataframe is expected where the lowest level

period : float

Seconds per sample

units : str

unit of the arg (D,s,ms,us,ns) denote the unit, which is an integer or float number.

relVel : bool

Output the lag in terms of the relative velocity change.

Returns

——-

DF : DataFrame

Original dataframe with the lag columns modified as specified.

CC_integration(DF, dx='index', intType='trapz')

Performs the integration of multiple columns witin a dataframe which is expected to contain row index levels ‘srcNo’, ‘recNo’ and ‘Time’. Each srcNo and recNo pair over time will be integrated and added back into the DF.

Parameters:

DF : DataFrame

Dataframe of the data to be integrated which must be row indexed by datetime64[ns].

dx : str (Default ‘index’)

If set to ‘index’ than integration is performed based on the time axis

intType : str (Default ‘trapz’)

The type of integration to use. trapz for trapezoid, cumsum for pure cummulative summation.

Returns

——-

DF_int : DataFrame

THe same dimensions and row indicies as DF containing the cumtrapz integration.

CC_to_K(DF)

Convert all Cross-correlation coefficients to decorrelation by applying the simple transform:

\[K = 1- CC\]

Parameters:

DF : DataFrame

Dataframe with multi level columns where the cross-correlation coefficients are expected to be named ‘CC’.

Returns:

DF_int : DataFrame

The dataframe is returned with only the data of the columns modified.

Data_CSV_dump(DF, fileName, colNames=None, indices=None, CCtoK=False, shiftCols=None, nthRow=None)

Dumps data from pandas dataframe to csv, intended for tikz plotting. Note, the following char will be removed from the column names ,’_[]% and any row of the selected colNames containing atleast one NaN will be filled with 0.

Parameters:

DF : DataFrame

DataFrame to be dumped

fileName : str

Name of the csv file saved to current working directory.

colNames : list

The columns to be dumped.

indices : slice

The slice object of indix values to be dumped.

CCtoK : bool (Default = False)

Convert all cross-correlation data to decorrelation. Expected column names as tuples with the last entry equal to ‘CC’ or ‘CC_mean’

shiftCols : list (Default = None)

List of columns to begin as zero (The first value will be subtracted from all)

nthRow : list (Default = None)

List of columns to begin as zero (The first value will be subtracted from all)

Data_atGT(DF, targetCols, outputCols, points, pointsCol, shiftCols=None)

Extracts first datapoint greater than a defined column value

Parameters:

DF : DataFrame

DataFrame to be dumped

targetCols : list

list of all target columns in DF

outputCols : list

list of output column names to use in output dataframe DF_out. Must be of equal length to targetCols.

points : list

list of points at which the first values > should be extracted from each entry in targetCols.

pointsCol : str

list of output column names to use in output dataframe DF_out. Must be of equal length to targetCols.

shiftCols : list (Default = None)

List of columns to begin as zero (The first value will be subtracted from all)

Returns:

df_trans : DataFrame

Output dataframe containing the requested points

TS_Time(DF, secPerSamp, traceSlice, resampleStr='1 us', csvDump=True, wdwPos=None, fileName='traces.csv')

Takes raw time series dataframe, allowing the slicing, resampling and re-indexing. Output times are in seconds.

Parameters:

DF : DataFrame

DataFrame to be modified.

secPerSamp : float

The sampling period or seconds per sample required to generate a time index for the dataframe.

traceSlice : slice

The slice object to extract from the DF.

resampleStr : str (default = ‘1 us’)

The resample string to reduce the size of each trace.

csvDump : bool (default = True)

Save data to traces.csv in pwd in the order, time [sec], trace1, trace2,….

wdwPos : list

List of [start, stop] window positions in no. of smaples

fileName : str (Default = ‘traces.csv’)

Name of output csv file.

Returns:

DF : DataFrame

Output dataframe.

wdwTimes : list

List of lists of the window positions in seconds

hdf_csv_dump(DB_fdl)

Dumps the processed databases to CC, PV, TShdrs to csv files. Note this function should be run in the run folder, not the database folder

—inputs— DB_fdl: relative or absolute location to the folder where all database files are located

run_dataLoad(DB_fdl)

Loads a previous processing session into memory ready for analysis.

• Inputs -

DB_fdl: input folder holding the expected databases in the form

‘DB_fld/’

• Outputs -

PV_df: Main database holding PV, and CC data TS_DB: Database of TS data PV_df_full: Database including all PV data, empty if original PV and TS

data was coincident already.

class pyCoda.data.data_import(TSfpath, PVloc, import_dtype, param=None)

Bases: object

This class handels the import and basic processing of all user imput data. The core data streams are the time series information (TS) and the corresponding Perturbation Vectors (PV)

Parameters:

TSfpath : str

Defines the relative or absolute location of the TS data folder

TSlocL : list (default = None)

List of the relative or absolute location of the TS data files

PVloc : str

Defines the relative or absolute location of the PV’s

Database : str

Defines a common hdf5 database name Database.h5

import_dtype: str

Defines several raw data types, “bin_par”: for data in a binary single trace per file and header data in .par files, “Shell_format”: all data in a single csv file (both PV and TS), ‘NoTShdrer_format’ “NoTShdrer_format”``.

notes

—–

The output of this class should be a single ``Database.h5`` database in the

run directory, containing all relevant data. All user defined parameters are

assigned to the attribues of the database head.

Examples

>>> import h5py
>>> # Reading the user defined parameters from the database attributes
>>> with h5py.File('Database.h5', 'r') as h5file:
>>>     print(dict(h5file.attrs.items()))

TSfiles()

This function lists the files in the TSfpath folder location reading all of the information contained within. The structure of this data is checked and the appropriate sub-function initiated based on the user defined parameter ‘import_dtype’

Parameters:

headerDB : DataFrame

Database of header information

TSdataList : list

List of TS data file relative locations

Returns:

headerDB : DataFrame or dict(DataFrames)

DataFrame of all header file information or dict of DataFrames. The structure should be index | srcNo | recNo | Time | “other header info”

read_finfd(file_loc)

Basic function to read files in folder and sort by numeric end digits Parameters ———- file_loc: str

Path to files.

TSfilesPar()

This function is intended to perform the TSfile function operations for a folder containing .par headerfiles and associated binary files.

Parameters:

List of all header .par and Binary files

Returns:

hdr_df Database of all header file information, multiple

index file name. Must contain mandatory columns

['srcNo', 'recNo', 'Time', 'Survey'].

TSdataL List of data files including relative location.

TODO:

• Add check for various header file types .txt ..etc

TSfilesPV_TS()

Loads in impuse response header data from a single folder. The expected format is the shell data structure.

Parameters:

self.TSlocL : list

List of all files within given folder

Returns:

df_hdr : DataFrame

Database of all header file information for each file. Must contain mandatory columns ['srcNo', 'recNo', 'Time', 'Survey'].

TSfilesCSIRO()

Loads in impulse response header data from multiple subfolders throughout a survey. Each subfolder should be of the format “sometext”YYYYMMDDHHMMSS. Within each subfolder are files for each source receiver pair.

Parameters:

self.TSlocL : list

List of all files within given

Returns:

header_dict : dict

Database of all header file information, multiple

index file name

TdeltaIdx : timedelta64[ns]

Time delta index of length equal to trace length of TS data

self.TSlocL : list of lists

Updated list of lists of each survey folders contents.

TSload(TShdrs)

Load the list of TS files TSflist found and output a matrix storing TS data columnweise.

Parameters:

TShdrs: DataFrame

A database of each file name containing the corresponding header. This is output from the function TSfiles.

TSflist: list

A list of the files within the TSfpath folder location. This is output from the function TSfiles.

Returns:

TSdataMtx : numpy array or None

A columnweise stored matrix of traces for a single receiver, or None if a multiple receiver survey is detected. In this case all TS data will be saved into a hdf5 database ‘TSdata.h5’.

TSsurveys(hdr_df)

Load TS data from a folder with sub-folders for each survey. The sub- folders should be named ‘sometext’(unique_number)’ (e.g. ‘survey20180312093545’). The individual csv files must be named ‘sometext’(unique_number)_(sourceNo)_(receverNo).(csv type formate) (e.g. ‘survey20180312093545_0001_01.csv’). A hdf5 database will be saved with the following group structure.

Database.h5 │ TSdata

│ └───survey20180312093545 │ Table └───survey20180312093555 │ Table : :

Each table includes header data from the csv files imported from the second line of the csv files as Date=12-03-2018; Time=09:35:45.833000; TracePoints=4096; TSamp=0.10000; TimeUnits=1.00000e-006; AmpToVolts=1.0000; TraceMaxVolts=5.0000; PTime=0.00000; STime=0.00000;

Parameters:

hdf_df : dict of DataFrames

A database of header information for each Time-series recorded. As a minimum must contain the first three columns of ‘srcNo’, ‘recNo’ and ‘time’. The remaining header info can be any order.

self.TSlocL : list of lists

List of lists of the files within each survey

Returns:

_ : None

None indicating that data is stored in a HDF5 file.

Notes

No longer is the TdeltaIdx added to each dataframe, thus I should figure out how best to store this information into the h5 file.

Examples

>>> import pandas as pd
>>> import h5py
>>> with h5py.File('Database.h5', 'a') as h5file:
>>>     TSsurvey =  pd.read_hdf(h5file, 'survey20180312093545') # basic load
>>>     # Load specific source-receiver pair for window between 11 and 500
>>>     TSsurvey =  pd.read_hdf(h5file, 'survey20180312093545',
            columns=[(1,1), (1,2)], # Load specific columns
            where = ['index>11', 'index<20']) # Load index 11 -> 20

TSloadBin(TShdrs)

Load in the binary file formate as expected from the ‘import_dtype’= ‘bin_par’.

TSloadPV_TS(TShdrs)

Read in time-series data and save to database

PVload()

Load Perturbation Vectors into a single database and store in ‘Database.h5’ within group ‘PVdata’.

Database.h5

│ PVdata

Table

Parameters:

self.PVloc : str

Path to PV data file/files

self.import_dtype : str

Indication of the data type, either ['.xls', 'bin_par', 'CSIRO'] or the ‘Shell_format’.

PVloadPV_TS()

Loads in the PV data from the shell format csv files

Parameters:self.TSfiles() : list of files of containing both PV and TS data

run_data_import()

Runs the class functions for importing the raw data, and stores this all within a database ‘Database.h5’

pyCoda.dataStore module

Created on Thu Mar 16 17:28:03 2017

@author: rwilson

class pyCoda.dataStore.utilities

Bases: object

A logical collection of functions for interacting with

static DB_pd_data_load(Database, group)

Loads in a pandas dataframe stored in group from the Database.

Parameters:

Database : str

Relative location of database

group : str

The expected group name

Returns:

group_df : DataFrame

The PV data stored in the group PVdata as a pandas dataframe

static hdf_csv_dump(DB_fdl)

Dumps the processed databases to CC, PV, TShdrs to csv files. Note this function should be run in the run folder, not the database folder

—inputs— DB_fdl: relative or absolute location to the folder where all database files are located

static run_dataLoad(DB_fdl)

Loads a previous processing session into memory ready for analysis.

• Inputs -

DB_fdl: input folder holding the expected databases in the form

‘DB_fld/’

• Outputs -

PV_df: Main database holding PV, and CC data TS_DB: Database of TS data PV_df_full: Database including all PV data, empty if original PV and TS

data was coincident already.

class pyCoda.dataStore.dataStore(param={}, PV_df=[], PV_df_full=[], TS_df=[], TS_DB=[], TShdrs=[], CC=[])

Bases: object

This class is intended to handel the storage of all data aquired and or generated during the processing.

pre_process()

check inputs are in the correct or expected format, process if required.

post_process()

check loaded DB is of the expected format, process if required.

checkSetup()

This functions checks the setup file to determine if any param have changed. If yes, the processing will be re-run, otherweise the saved datebases will be loaded. return True if change is detected

to_hdf5(DB_tbl, df, tbl_name, form)

Save expected df to hdf5 database

from_hdf5(DB_tbl, tbl_name)

Save expected df to hdf5 database

from_sql_DB(tbl_name, DB_egn)

Load data from the sql_DB uponsd

from_pkl(fname)

Load pickel files fname: file name rel or abs path

to_pkl(data, fname)

Save pickel files data: data to pickel fname: file name rel or abs path

hdf_to_csv(hdf_DB, tbl_name)

Save hdf DB to csv hdf_DB: HDF5 database rel of abs path and name tbl_name: Name of table in database

run_dataStore()

run the expected data storage workflow

run_dataLoad()

run the expected data loading workflow

pyCoda.dispCWI module

pyCoda.dispCWI_DB module

class pyCoda.dispCWI_DB.dispCWI_DB(param, DB, TS_DB, PV_full)

Bases: object

This class handels the interactive ploting of CC data stored within a single database.

wdw_lag_col()

Extracts the wdw pos from DB column names

CC_labels(tuples)

Handels the assignment of figure titles and axis labels. Basic functionality, checks if variables is assigned in param, if not a a default is assigned. tuples: tuples of (lag, self.self.wdw_pos)

num_in_str(num, string, pattern)

Match an exact numnber in a string based on a pattern

TS_plot_wdw(ax, wd_pos)

This private function handels the general plotting of the correlation window considered. for each window in wd_pos Inputs: —— wd_pos: sample number of beginning of window TODO:

plot_DB()

Generate Interactive plot of DB

pyCoda.postProcess module

Created on Mon Apr 24 16:17:07 2017

@author: rwilson

class pyCoda.postProcess.post_utilities

Bases: object

A collection of post processing utility functions

detect_most_linear(x, y, m, smooth=False, wdw_wdth=25, poly_ord=3)

Finds the most linear portion of a line via Inputs: —— x : Series

y-axis array of values

y : Serues

y-axis array of values

m : Length of most linear portion of line (x,y) smooth : bool

Smooth the curve before search

detect_peaks(x, mph=None, mpd=1, threshold=0, edge='rising', kpsh=False, valley=False, show=False, ax=None)

Detect peaks in data based on their amplitude and other features.

Parameters:

x : 1D array_like

data.

mph : {None, number}, optional (default = None)

detect peaks that are greater than minimum peak height.

mpd : positive integer, optional (default = 1)

detect peaks that are at least separated by minimum peak distance (in number of data).

threshold : positive number, optional (default = 0)

detect peaks (valleys) that are greater (smaller) than threshold in relation to their immediate neighbors.

edge : {None, ‘rising’, ‘falling’, ‘both’}, optional (default = ‘rising’)

for a flat peak, keep only the rising edge (‘rising’), only the falling edge (‘falling’), both edges (‘both’), or don’t detect a flat peak (None).

kpsh : bool, optional (default = False)

keep peaks with same height even if they are closer than mpd.

valley : bool, optional (default = False)

if True (1), detect valleys (local minima) instead of peaks.

show : bool, optional (default = False)

if True (1), plot data in matplotlib figure.

ax : a matplotlib.axes.Axes instance, optional (default = None).

Returns:

ind : 1D array_like

indeces of the peaks in x.

Notes

The detection of valleys instead of peaks is performed internally by simply negating the data: ind_valleys = detect_peaks(-x)

The function can handle NaN’s

See this IPython Notebook [R112].

References

[R112]

(1, 2) http://nbviewer.ipython.org/github/demotu/BMC/blob/master/notebooks/DetectPeaks.ipynb

Examples

>>> from detect_peaks import detect_peaks
>>> x = np.random.randn(100)
>>> x[60:81] = np.nan
>>> # detect all peaks and plot data
>>> ind = detect_peaks(x, show=True)
>>> print(ind)

>>> x = np.sin(2*np.pi*5*np.linspace(0, 1, 200)) + np.random.randn(200)/5
>>> # set minimum peak height = 0 and minimum peak distance = 20
>>> detect_peaks(x, mph=0, mpd=20, show=True)

>>> x = [0, 1, 0, 2, 0, 3, 0, 2, 0, 1, 0]
>>> # set minimum peak distance = 2
>>> detect_peaks(x, mpd=2, show=True)

>>> x = np.sin(2*np.pi*5*np.linspace(0, 1, 200)) + np.random.randn(200)/5
>>> # detection of valleys instead of peaks
>>> detect_peaks(x, mph=0, mpd=20, valley=True, show=True)

>>> x = [0, 1, 1, 0, 1, 1, 0]
>>> # detect both edges
>>> detect_peaks(x, edge='both', show=True)

>>> x = [-2, 1, -2, 2, 1, 1, 3, 0]
>>> # set threshold = 2
>>> detect_peaks(x, threshold = 2, show=True)

MLTWA_calc(TS_DB, after_FB, Ewdth=None, wd_shift=0, ref_trace=0, mph=None, mpd=None, threshold=0, R1_Sign=False, grad_period=-50, verbose=False)

Apply Multi Lapse-Time Window Analysis on the input database time -series. Additional parameters are calculated such as the B0 or ratio of R1 to R2 as well as their gradient difference.

Parameters:

TS_DB : DataFrame

Time-series database in cronological order.

after_FB : DataFrame.index

Index of TS_DB after which the S-wave max value will be found

Ewdth : list, optional (default = [len(TS_DB)//16,len(TS_DB)//16,len(TS_DB)//4]

The first two energy window widths in sample points

wd_shift : int

Shift parameter of the start of windows

ref_trace : int

The trace to use as a reference for calculation of R2

mph : {None, number}, optional (default = None)

detect peaks that are greater than minimum peak height.

mpd : positive integer, optional (default = 1)

detect peaks that are at least separated by minimum peak distance (in number of data).

threshold : positive number, optional (default = 0)

detect peaks (valleys) that are greater (smaller) than threshold in relation to their immediate neighbors.

R1_Sign : bool (Default False)

Apply sign change to R1.

grad_period : int (Default False)

The number of periods to use when calculating the gradient of ratios R1 and R2.

verbose: bool, optional (default = False)

Provide a details output

Returns:

DB_MLTWA : DataFrame

Containing all of the MLTWA data with index equal to third expected Time column of the input TS_DB.

dict_MLTWA: dictcontaining parameters pertaining to the MLTWA processing.

-R1: Ratio of log_10 E_1(t_i)/ E_3(t_i) -R2: Ratio of log_10 E_1(t_u)/ E_1(t_i) -E1: Integrated energy early S-wave -E2: Integrated energy mid S-wave -E3: Integrated energy late S-wave -E_wdw_pos: list of start, mid, end of E1,E2,E3 -idx_break: list detected peaks in search of the S-wave arrival

verbose: bool, optional (default = False)

Provide a detailed output

Examples

>>> import postProcess as pp
>>> idx = pp.post_utilities.MLTWA_calc(TScut_DB_in, after_FB = 0.00008,
                                       wd_shift=-500, mph=None, mpd=12,
                                       threshold=0, verbose=True)

PV_segmentation(PV_df, Segments, targets, indexName='index', shiftCols=None, verbose=False)

Apply a range of data parameterisation methods from segments of input data. Note, any row with a nan will be removed before segmentation.

Parameters:

PV_df : DataFrame

Time-series database in cronological order.

Segments: list

A list of indicies at which between which the input PV_df will be segmented and paramterised.

targets: list

List of column names in PV_df.

indexName : Str (Default ‘index’)

The index in which the Segments are defined.

shiftCols : list (Default = None)

List of columns to begin as zero (The first value will be subtracted from all)

verbose: bool, optional (default = False)

Provide a detailed output

Returns:

dict_Segments: dictcontaining parameters for each segment

-Sigma: Sigma of each segment -R2: R2 of each segment -Mean: mean of each segment -Skewedness: mean of each segment

Examples

>>> import postProcess as pp

>>> seg_list = [[4, 19.5], [21.74, 37.2], [39.68, 55.29]] # In Hours

>>> DF_Segments = pp.post_utilities.PV_segmentation(PV_df_in, seg_list,

targets = [‘Pore pressure| [MPa]’,’R1’,’R2’])

TS_FBP(TS_DB, noiseWD, threshold=1, threshold_shift=0, mpd=1, verbose=False)

This function is intended to perform first break picking

Parameters:

TS_DB : DataFrame

Columnweise DataFrame of TS data, index of time expected

noiseWD : int

Window width in TS_DB index which is expected to be only noise in number of sample points.

threshold : int/float

Percentage of the noise standard deviation which will define the detection threshold.

threshold_shift : float

A shift to the (threshold * noiseStd)

mpd : int

Minimum poit distance in number of samples

verbose : bool

If True, the interactive plotting of picks will be made

Returns:

idx_break : int

index of the first break detection.

TS_interactive(TS_DB, idx=None, threshold=False, noiseWD=False, wdws=None)

Interactive plot of time-series data with ability to plot detection parameters for analysis.

Parameters:

TS_DB : dataframe

Database of all input time series in each columns

idx : int (default None)

The index of the detected peak

threshold : list (default False)

Thresholds for each trace in terms of amplitude

noiseWD : int (default False)

The noise zeroed window length

wdws : list (default None)

A list of window index start stop positions e.g. [[sta, stp]…].

CC_ch_drop(CC_DB, channels=None, errors='raise')

Drops channels from standard CCdata dataframe

Parameters:

CC_DB : dataframe

Database containing in the first two levels src and rec numbers

channels : int (default = None)

the channels to remove from the dataframe, either a list of channels in which case both source and receivers will be dropped, or a list of channel pairs in which case only the defined pairs will be dropped.

errors : str (dfault = ‘raise’)

Raise error in attempted drop does not exist, ‘ignore’ to surpress the error.

calcSNR(TSsurvey, Noise_channels, all_channels, wdws, noiseCutOff=0, inspect=False)

Determines the channels which are above a certain SNR.

Parameters:

TSsurvey : dataframe

Single survey dataframe of traces

Noise_channels : list

the channels on which the noise will be estimated

all_channels : list

All channels numbers

wdws : list

The windows in samples points at which the SNR is calculated

noiseCutOff: float

The threshold of SNR in Db to filter

inspect: bool (default = False)

Inspect the traces which are noted as noise.

Returns

——-

noiseyChannels : float

The noisey channels

smooth(x, window_len=11, window='hanning')

smooth the data using a window with requested size.

This method is based on the convolution of a scaled window with the signal. The signal is prepared by introducing reflected copies of the signal (with the window size) in both ends so that transient parts are minimized in the begining and end part of the output signal.

input:

x: the input signal window_len: the dimension of the smoothing window; should be an odd integer window: the type of window from ‘flat’, ‘hanning’, ‘hamming’, ‘bartlett’, ‘blackman’

flat window will produce a moving average smoothing.

output:

the smoothed signal

example:

t=linspace(-2,2,0.1) x=sin(t)+randn(len(t))*0.1 y=smooth(x)

see also:

numpy.hanning, numpy.hamming, numpy.bartlett, numpy.blackman, numpy.convolve scipy.signal.lfilter

TODO: the window parameter could be the window itself if an array instead of a string NOTE: length(output) != length(input), to correct this: return y[(window_len/2-1):-(window_len/2)] instead of just y.

PV_time_shift(PVdata, PV_time_col, PV_time_col_unit, PVstart_date_time)

Adjusts the Time Stamp column in the PVdata frame.

Parameters:

PVdata : DataFrame

The dataframe on which operations are performed

PV_time_col : str

The name of the PV time column to be processed.

PV_time_col_unit : str

Unit of the time column PV_time_col.

PVstart_date_time : str

The YYYY-MM-DD ..etc string defining the origin of the PV_time_col.

Examples

>>> # Determine the origin time of the PVdata
>>> origin = pd.to_datetime('20180212180703') - pd.Timedelta(43.6725, unit='D')
>>> # Shift the PVdata to the new origin
>>> pp.post_utilities.PV_time_shift(PVdata, 'Time(Days)', 'D', origin)
>>> # Save the new PVdata back to the HDF5 database
>>> dt.utilities.DB_pd_data_save('Ultrasonic_data_DB/Database.h5', 'PVdata', PVdata)

class pyCoda.postProcess.postProcess(param, PV_df, CC, TS=[])

Bases: object

This class is intended to handel post processing of a database of CC and PV data.

PV_CC_join()

Combine both PV and CC data into a single dataset based on measurement number

TS_DB()

Convert TS matrix into a DataFrame with axis index in time

postProcess_run()

Perform expected Post Processing

pyCoda.pre_process module

pyCoda.pycker module

pyCoda.runCWI module

pyCoda.userInput module

Created on Thu Jan 12 13:37:48 2017

@author: rwilson

class pyCoda.userInput.utilities

Bases: object

Collection of functions tools applicable to user interaction.

static query_yes_no(question, default='yes')

Ask a yes/no question via raw_input() and return their answer.

“question” is a string that is presented to the user. “default” is the presumed answer if the user just hits <Enter>.

It must be “yes” (the default), “no” or None (meaning an answer is required of the user).

The “answer” return value is True for “yes” or False for “no”.

class pyCoda.userInput.userInput(fileIn)

Bases: object

This class is intended to handle the direct input general information for the purpose of processing, display and documentation. A .txt file is expected with a basic, variable and value format. % comments are ignored.

var = val % val can be string or numeric input

All identified variables are saved into a dictionary of the format dic = {var1: val1, var2: val2 }

This class is also intended to compare the initial user input info with the previous run, and return re_run as False if the same inputs are given.

Parameters:fileIn Location of user input txt file

Notes

The accepted inputs are as follows:

Setup paramaters: import_raw : Request Raw data to be imported (Default: False)

re_run : Force the reprocessing, even if no changes found in input file.

import_dtype : The data type to import(bin_par, Shell_format, ‘CSIRO’.

survey_type : multiple (Default) if multiple src/rec pairs in subfolder structure

is expected. single if a single src/rec pair is expected with no subfolders.

TSstart_date : str

The start of TS acquistion in format YYYY-MM-DD. Only required if no absolute time data available in TS header information.

TSloc : location of folder/file containing the time series data.

PVloc : location of folder/file containing the perturbation data.

loadDB : If True, no attempt to reload raw data into database will be made

——————- Pre-processing inputs ——————- PVstart_date_time : (Default None) The time YYYY-MM-DD HH:MM:SS.sssss

the first PV measurement was made, in sync with TS time.

PV_time_col : (required) Name of time column to match in PV data

PV_time_col_unit : (Default = ‘D’) the time unit of the PV column to match,

see pd.to_datetime

sampNo : The number of sample points in a single trace recording, only

required as input if not found in TS header information.

sampFreq : The sampling frequence of TS data, [samp/sec]

——————- Cross-correlation parameters ——————- sig : bool

If True is given then a spectral significance test will be made for each correlation performed. This is an expensive calculation so will require considerable time to perform. See cross-correlation

lagOverlap : bool (Default True) UNDER DEVELOPMENT

Allow overlap between rolling lag values (i.e. 1-3, 2-4). If set to False, only none overlapping lags will be calculated (e.g. 1-3, 3-5).

Eng_ref : int (Default None)

Indicating the trace/survey to which all subsequent relative energy calculations will be referenced to.

STA_meas : int/str (Default False)

The first survey to include in the correlation processing, if false then no start will be set. Note: only for survey_type multiple, a date string in the format YYYY-MM-DD HH:MM:sec.msec must be given, which corresponds to the survey folder number.

sta_wdws : int (Optional)

If wdwPos and ww_ol as given then the user can also define the start of the series of overlapping windows.

end_wdws : int (Optional)

If wdwPos and ww_ol as given then the user can also define the end of the series of overlapping windows.

wdwPos : (int, Default = False) The start position of the windows in trace sample numbers,

if more than one is provided then ww must be of equal length. If not provided then both ww and ww_ol should be given.

ww : (int) Width of each window in trace sample numbers,

if more than one is provided then wdwPos must also be of equal length.

ww_ol : (int) (Default False)

The percentage window overlap. The max number of windows of ww lenght which fit within a trace will be calculated. This will be ignored if wdwPos is provided.

taper : bool

Apply a tukey window taper to each correlation window

taperAlpha : float (Default 0.1)

The alpha of the window taper to be applied.

CC_folder : (str, Default = ‘CCprocessed’)

The name of the folder within which the CC processed data will be saved.

——————- Display parameters ——————-

disp_DB : bool (Default True)

If set to true, some portion of the processed database will be plotted to screen at the end of the Class runCWI.

fread()

is_number(s)

checkSetup(check)

This functions checks the setup file to determine if any param have changed. If yes, the processing will be re-run, otherweise the saved datebases will be loaded. return True if change is detected, False if any critical param are found and None if only non-critical param change

sys = <module 'sys' (built-in)>