发一个复习数科python coding challenge复习的总结

跳槽求职，总结了一些python data wrangling的常见function。主要是panda，欢迎大家补充！

Functions
Create function for lambda
Example 1:
def myfunc(position, result_check):
    if result_check==True and position<=3:
        myvalue=5
    elif result_check==True and position<=5:
        myvalue=4
    elif result_check==True and position<=10:
        myvalue=3
    elif result_check==True and position>10:
        myvalue=2
    else:
        myvalue=1
return myvalue
fb_search_results['rating'] = fb_search_results.apply(lambda x: myfunc(x['position'], x['result_check']), axis=1)
Example 2:
facebook_posts['is_spam'] = facebook_posts['is_spam'].apply(lambda x: 1 if x == True else 0)
Example 3:
fb_search_results['result_check'] = fb_search_results.apply(lambda x: x.query.lower() in x.notes.lower(), axis=1)   
Example 4:
result['business_type'] = result['business_name'].apply(
lambda x: 'school' if 'school' in x.lower()
    else 'restaurant' if 'restaurant' in x.lower()
    else 'cafe' if 'cafe' in x.lower() or 'coffee' in x.lower()
else 'other')
# create new column with dict
df.replace({"col1": di})

Order by asc/ desc
df.sort_values(by='col1', ascending=False)

Not in
~customers['id'].isin(name_ls)

In
customers['id'].isin(name_ls)

Or
(df['A'] == 3) | (df['B'] == 7)

Top
Df.head(5)

Null
Df.col.Isna()
Df.col.notnull()

Unstack
DataFrame.unstack(level=- 1, fill_value=None)
Pivot a level of the (necessarily hierarchical) index labels.
Returns a DataFrame having a new level of column labels whose inner-most level consists of the pivoted index labels.
If the index is not a MultiIndex, the output will be a Series (the analogue of stack when the columns are not a MultiIndex).
Parameters
Level int, str, or list of these, default -1 (last level)
Level(s) of index to unstack, can pass level name.
fill_value int, str or dict
Replace NaN with this value if the unstack produces missing values.

Contains
Series.str.contains(pat, case=True, flags=0, na=None, regex=True)
Test if pattern or regex is contained within a string of a Series or Index.
Return boolean Series or Index based on whether a given pattern or regex is contained within a string of a Series or Index.
Parameters
Pat str
Character sequence or regular expression.
Case bool, default True
If True, case sensitive.
Flags int, default 0 (no flags)
Flags to pass through to the re module, e.g. re.IGNORECASE.
Na scalar, optional
Fill value for missing values. The default depends on dtype of the array. For object-dtype, numpy.nan is used. For StringDtype, pandas.NA is used.
Regex bool, default True
If True, assumes the pat is a regular expression.
If False, treats the pat as a literal string
Example 1: airbnb_search_details['amenities'].str.contains('beach', case=False)

Offset
df.col1.shift(-1)

Row number
Df.reset_index()

Fill NA
Df.fillna(0)

Idxmin
DataFrame.idxmin(axis=0, skipna=True)
Return index of first occurrence of minimum over requested axis.
NA/null values are excluded.
Parameters
Axis {0 or ‘index’, 1 or ‘columns’}, default 0
The axis to use. 0 or ‘index’ for row-wise, 1 or ‘columns’ for column-wise.
skipnabool, default True
Exclude NA/null values. If an entire row/column is NA, the result will be NA.

Replace
relative_variance_hr.replace({"PWCOUNTY_name": county_name}).copy()

Rank
df1['rank'] = df1['msg_count'].rank(ascending=False)
- method{‘average’, ‘min’, ‘max’, ‘first’, ‘dense’}, default ‘average’
How to rank the group of records that have the same value (i.e. ties):
•        average: average rank of the group
•        min: lowest rank in the group
•        max: highest rank in the group
•        first: ranks assigned in order they appear in the array
•        dense: like ‘min’, but rank always increases by 1 between groups
- pct, default False
Whether or not to display the returned rankings in percentile form

Append
Df1.append([df2, df3])

Drop_duplicates
DataFrame.drop_duplicates(subset=None, keep='first', inplace=False, ignore_index=False)
Return DataFrame with duplicate rows removed.
Considering certain columns is optional. Indexes, including time indexes are ignored.
Parameters
Subset column label or sequence of labels, optional
Only consider certain columns for identifying duplicates, by default use all of the columns.
Keep {‘first’, ‘last’, False}, default ‘first’
Determines which duplicates (if any) to keep. - first : Drop duplicates except for the first occurrence. - last : Drop duplicates except for the last occurrence. - False : Drop all duplicates.
Inplace bool, default False
Whether to drop duplicates in place or to return a copy.
ignore_index bool, default False
If True, the resulting axis will be labeled 0, 1, …, n - 1.
New in version 1.0.0.
Example 1: df.drop_duplicates(subset=['brand', 'style'], keep='last')

Groupby
After groupby, you can add  'max', 'mean', 'median', 'min', 'count', ‘prod’ (i.e. taking product 3*4), 'cumcount', 'cummax', 'cummin', 'cumprod', 'cumsum', 'fillna', 'filter', 'nunique', 'pct_change', 'quantile', 'rank',  'sum', ‘size’, ‘nlargest(n)’
# agg
Example 1: result = result.groupby('post_date').agg({'is_spam': ['sum', 'count']}).reset_index()
Example 2: df.groupby('A').agg({'B': ['min', 'max'], 'C': 'sum'})
Example 3: count_facilities = los_angeles_restaurant_health_inspections.groupby(["facility_zip"])['facility_id'].agg(no_facilities='nunique', no_inspections='count').reset_index()
# apply: use it with lambda function
# diff
Periods: int, default 1
Periods to shift for calculating difference, accepts negative values.
Axis: {0 or ‘index’, 1 or ‘columns’}, default 0
Take difference over rows (0) or columns (1).
# transform
DataFrameGroupBy.transform(func, *args, engine=None, engine_kwargs=None, **kwargs)
Call function producing a like-indexed DataFrame on each group and return a DataFrame having the same indexes as the original object filled with the transformed values
Parameters
F function
Function to apply to each group.
Can also accept a Numba JIT function with engine='numba' specified.
If the 'numba' engine is chosen, the function must be a user defined function with values and index as the first and second arguments respectively in the function signature. Each group’s index will be passed to the user defined function and optionally available for use.
Changed in version 1.1.0.
*args
Positional arguments to pass to func.
Engine str, default None
•        'cython' : Runs the function through C-extensions from cython.
•        'numba' : Runs the function through JIT compiled code from numba.
•        None : Defaults to 'cython' or globally setting compute.use_numba
engine_kwargs dict, default None
•        For 'cython' engine, there are no accepted engine_kwargs
•        For 'numba' engine, the engine can accept nopython, nogil and parallel dictionary keys. The values must either be True or False. The default engine_kwargs for the 'numba' engine is {'nopython': True, 'nogil': False, 'parallel': False} and will be applied to the function
New in version 1.1.0.
**kwargs
Keyword arguments to be passed into func.

Pivot_table
pandas.pivot_table(data, values=None, index=None, columns=None, aggfunc='mean', fill_value=None, margins=False, dropna=True, margins_name='All', observed=False)
Create a spreadsheet-style pivot table as a DataFrame.
The levels in the pivot table will be stored in MultiIndex objects (hierarchical indexes) on the index and columns of the result DataFrame.
Parameters
Data DataFrame
Values column to aggregate, optional
Index column, Grouper, array, or list of the previous
If an array is passed, it must be the same length as the data. The list can contain any of the other types (except list). Keys to group by on the pivot table index. If an array is passed, it is being used as the same manner as column values.
Columns column, Grouper, array, or list of the previous
If an array is passed, it must be the same length as the data. The list can contain any of the other types (except list). Keys to group by on the pivot table column. If an array is passed, it is being used as the same manner as column values.
Aggfunc function, list of functions, dict, default numpy.mean
If list of functions passed, the resulting pivot table will have hierarchical columns whose top level are the function names (inferred from the function objects themselves) If dict is passed, the key is column to aggregate and value is function or list of functions.
fill_value scalar, default None
Value to replace missing values with (in the resulting pivot table, after aggregation).
Margins bool, default False
Add all row / columns (e.g. for subtotal / grand totals).
Dropna bool, default True
Do not include columns whose entries are all NaN.
margins_name str, default ‘All’
Name of the row / column that will contain the totals when margins is True.
Observed bool, default False
This only applies if any of the groupers are Categoricals. If True: only show observed values for categorical groupers. If False: show all values for categorical groupers.
Changed in version 0.25.0.
Example 1: pd.pivot_table(df, values='D', index=['A', 'B'], columns=['C'], aggfunc=np.sum, fill_value=0)
Example 2: pd.pivot_table(df, values=['D', 'E'], index=['A', 'C'], aggfunc={'D': np.mean, 'E': [min, max, np.mean]})
Rename multindex table
Example 1:
pivot_df.columns = pivot_df.columns.droplevel(0)
pivot_df.columns.name = None
pivot_df.columns= pivot_df.columns.astype(str)
pivot_df = pivot_df.reset_index()
result = pivot_df.rename(columns = {'Entire home/apt':'apt_count','Private room':'private_count','Shared room':'shared_count'})

Merge
pd.merge(left, right, how='inner', on=None, left_on=None, right_on=None,
         left_index=False, right_index=False, sort=True,
         suffixes=['_x', '_y'], copy=True, indicator=False,
         validate=None)
sort bool, default False
Sort the join keys lexicographically in the result DataFrame. If False, the order of the join keys depends on the join type (how keyword).
Copy bool, default True
If False, avoid copy if possible.
Indicator bool or str, default False
If True, adds a column to the output DataFrame called “_merge” with information on the source of each row.
Validate str, optional
If specified, checks if merge is of specified type.
•        “one_to_one” or “1:1”: check if merge keys are unique in both left and right datasets.
•        “one_to_many” or “1:m”: check if merge keys are unique in left dataset.
•        “many_to_one” or “m:1”: check if merge keys are unique in right dataset.
•        “many_to_many” or “m:m”: allowed, but does not result in checks.

String manipulation
# string extract by location
Df.col.str[6, 2]
# string extract by content
Df.col.str.contains()
# add string
‘text1’+’text2’
# split
Pd.DataFrame(Df.col.Split(‘,’).tolist())
Calculation Operation
# round
df2['pay_emp'] = round(df2['pay_emp'], 2)
# others
abs(), np.sqrt(), np.std(), power: 5**3 = 125
Data Type  
Df.col.astype({'col1': 'int32'})
Calculation Date
# convert datetime
from datetime import datetime
datetime.now().year
# format
pd.to_datetime(df['activity_date'], format='%Y-%m-%d' ).dt.strftime('%Y-%m-%d')
# delta
fb_comments_count['created_at'] >= pd.to_datetime('2020-02-10') - timedelta(days=30)
# find year/month/day
X.INCDTTM = pd.to_datetime(X.INCDTTM)
X.INCDTTM.dt.year
# between
merged['date'].between('2020-03-01', '2020-03-31')]

Rename
count = count.rename({0: 'count'}, axis='columns').copy()

Regax
result['business_name'].replace('[^a-zA-Z0-9 ]','',regex=True)

Tips and Tricks
# remember to sort see if we need to include name/ product that has zero records.





. ----















.1point3acres