📣 黑五价提前开抢|VIP一年通行证5折,蓝莓、Offer多多同步特惠
楼主: 宋龙龙96
跳转到指定楼层
上一主题 下一主题
收起左侧

Big data programming

🔗
卡卡的门 2019-2-26 15:56:09 | 只看该作者
本楼:
全局:
非常感谢!
回复

使用道具 举报

🔗
 楼主| 宋龙龙96 2019-2-27 08:49:35 | 只看该作者
全局:
Lecture 5
SCALA
Immutability:
Why?
▸ Immutable objects are automatically thread-safe
(you don’t have to worry about object being changed by another thread)
▸ Compiler can reason better about immutable values -> optimization


Lists

▸ Lists are immutable (= contents cannot be changed)
▸ List[String] contains Strings
val lst = List("b", "c", "d")
lst.head // “b”
lst.tail // List(“c”, “d”)
val lst2 = "a" :: lst // cons operator

Arrays
▸ Lists are immutable, arrays are mutable
val a = Array("Java", "rocks")
a(0) = "Scala";



Covariance
Lists are covariant, arrays are invariant
// compiler error
val array : Array[Any] = Array(1, 2, 3);
// ok
val list : List[Any] = List(1, 2, 3);
Everything in scala is an object:
0 to 2 === 0.to(2)
▸ x – 1 === x.-(1)
Every operation is a method call
▸ map containsKey ‘a’ === map.containsKey(‘a’)

https://github.com/yingmao/ScalaExamples/blob/master/ListSample/ListSample.scala


Currying)指的是将原来接受两个参数的函数变成新的接受一个参数的函数的过程。新的函数返回一个以原有第二个参数为参数的函数。
http://www.runoob.com/scala/currying-functions.html

https://github.com/yingmao/ScalaExamples/blob/master/ListFlat/CaseMatchSample.scala

SCALA static

class A
int B
static int c
-------------------------
A1=new A()
A2=new A()
AI,A2have diff B but share C(cz static)
CIS

(year, maker)
clean data first
decide which key to decided to count?


PRoject:  which zones comfortable?


slides and reading assignment.



回复

使用道具 举报

🔗
xiao66xiang 2019-2-27 11:33:23 | 只看该作者
全局:
请问这是什么课呀
回复

使用道具 举报

🔗
 楼主| 宋龙龙96 2019-2-28 04:06:14 | 只看该作者
全局:
xiao66xiang 发表于 2019-2-27 11:33
请问这是什么课呀

大数据编程课~
回复

使用道具 举报

🔗
xiao66xiang 2019-2-28 08:33:05 | 只看该作者
全局:

是公开课吗?还是你自己学校的?
回复

使用道具 举报

🔗
 楼主| 宋龙龙96 2019-2-28 08:40:49 | 只看该作者
全局:
xiao66xiang 发表于 2019-2-28 08:33
是公开课吗?还是你自己学校的?

学校的课,资料是自己查的。
回复

使用道具 举报

🔗
 楼主| 宋龙龙96 2019-3-6 09:23:15 | 只看该作者
全局:

threads

Threads are used to execute code concurrently
▸ Threads contains resources that they are shared
▸ Synchronization help guarantee:
      ▸ Correct ordering▸ Visibility▸ Data consistency
▸ Problems?
   ▸ Hard to solve deadlocks▸ Hard to maintain data consistency▸ Hard to ensure liveness
dead lock: T1&T2


T1-----> A
lock        ^
V           lock
B<-------T2

T1  inti
1~10
T2 read i
sum=+i
Producer-Consumer example: 每次计算不一样。
Setup– A shared memory buffer– Producer puts objects into the buffer– Consumer reads objects from the buffer• ProducerConsumerTest.java, UnsyncBuffer.java• Problem:– producer can over-produce, consumer can over-consume (example of race condition)– Need to synchronize (coordinate) the processes

Disadvantages of synchronization
• Disadvantages:– Synchronization is error-prone– Synchronization blocks threads and takes time– Improper synchronization results in deadlocks– Creating a thread is not a low-overhead operation– Too many threads slow down the system

empty lock
dequeqe :run
59:
java 's methods have a lot of distan.
share variable(why: we share a lock)
An Akka/Scala concurrent solution
Master actor:  difeerent 听录音:


• Initiates the process• sends tasks to worker actors• collects the total size (sum them up)• keeps track of pending tasks
-----------------------------------------------------
Q8: scalaa 10 examples  three schudling
sLide 80%
coding
why we need move r data  
map. reduce physical machine .  ,,,,听录音

design a map -reduce . write 伪代码+描述去解释查 slides. [give your design]
map1 produce XX map2 need to use them

Q8 read some code( understand code)

why schduling ? T/false
round-robin vs
single process

STF is optimal!有简单计算。看那页PPT


capacity scheduler:
Q1 80%
q1 20% soft limit


domimt one question from this.
-------------
scala froM one of example  5%
try to fix it  let me know what does the progam do


Dear professor Ying Mao,


I sent a lab missing some part. My teammate Shuyan Liu







回复

使用道具 举报

🔗
 楼主| 宋龙龙96 2019-3-28 05:33:49 | 只看该作者
全局:
您好!
本帖隐藏的内容需要积分高于 2000 才可浏览
您当前积分为 0。
使用VIP即刻解锁阅读权限或查看其他获取积分的方式
游客,您好!
本帖隐藏的内容需要积分高于 2000 才可浏览
您当前积分为 0。
VIP即刻解锁阅读权限 或 查看其他获取积分的方式
Popular Interview Questions from Top Companies
回复

使用道具 举报

🔗
 楼主| 宋龙龙96 2019-4-3 08:56:30 | 只看该作者
全局:
spark: memory    hadoop: storage(HDfs output)
spark: RDD use mem amap
exam:
R--> P1P2(linear,dont need output assume all data inmem,X needoutput =X loop优点-->W

Exam: draw RDD graph!
if log2 die?? NBA log-->log1-->?log2( we X lose anything. X take any data from HDFS)
regenerate log2

reduce the

file name ,how large file .

1' sc is the basic  of RDD


2. worker story data in memory (in those cache)share to the driver/ built those RDD(of whole system)
worker chache1 wor... (transmit the resultsto)==> driver

4 driver have all result ( but combiner only have one result)


回复

使用道具 举报

🔗
 楼主| 宋龙龙96 2019-5-9 10:08:29 | 只看该作者
全局:
Spark RDD and Spark SystemRDD Lineage

RDD Dependency
Spark vs MapReduce: Data Flow
--------------------------------
Three ways to create a RDD
▸ From a file or set of files
▸ From data in memory
▸ From another RDD
---------------------------
RDD Operation:
Two types of operations Transformations:
▸ Define a new RDD based on current RDD(s)
▸ Actions: return values
------------------------------
RDD Transformations
Set of operations on a RDD that define how they should be transformed
▸ As in relational algebra, the application of a transformation to an RDD yields a new RDD (because RDD are immutable)
▸ Transformations are lazily evaluated, which allow for optimizations to take place before execution
▸ Examples: map(), filter(), groupByKey(), sortByKey(), etc.
--------------------
Q7: Mesos, HBase, Hive, Pig
Q7: System Design

DStream – sequence of RDDs representing a stream of data
‣ Twitter, HDFS, Kafka, Flume, ZeroMQ, Akka Actor, TCP sockets
‣ Transformations – modify data from on DStream to another
‣ Standard RDD operations – map, countByValue, reduce, join, …
‣ Stateful operations – window, countByValueAndWindow, …
‣ Output Operations – send data to external entity
‣ saveAsHadoopFiles – saves to HDFS
‣ foreach – do anything with each batch of result






回复

使用道具 举报

您需要登录后才可以回帖 登录 | 注册账号
隐私提醒:
  • ☑ 禁止发布广告,拉群,贴个人联系方式:找人请去🔗同学同事飞友,拉群请去🔗拉群结伴,广告请去🔗跳蚤市场,和 🔗租房广告|找室友
  • ☑ 论坛内容在发帖 30 分钟内可以编辑,过后则不能删帖。为防止被骚扰甚至人肉,不要公开留微信等联系方式,如有需求请以论坛私信方式发送。
  • ☑ 干货版块可免费使用 🔗超级匿名:面经(美国面经、中国面经、数科面经、PM面经),抖包袱(美国、中国)和录取汇报、定位选校版
  • ☑ 查阅全站 🔗各种匿名方法

本版积分规则

>
快速回复 返回顶部 返回列表