For me, I split all the talks I listened this time in QCon, into 3 categories:
Tech Architecture Optimization
Tidy First
Scaling Bottleneck
System Design Concept
Microservices Patterns
Orchestration vs Choreography
System Design Implementation
Adopting Continuous Deployment
Distributed In-Process Cache
Leveraging Determinism
Service Mesh
DynamoDB in Amazon
CosmosDB in Azure
Event Processing Service in Doordash
Blob Storage Service in Dropbox
The categories are ordered from abstraction to concretization. I know most of the engineers like me are more interested in the low level of the implementations. And yes, some talks here(like the Determinism one is really really good), however, this time the first two categories’ talk gave me the biggest influence. I think listening to these talks help me to understand the career path better. Sometimes people have to view things out of the box.
I really appreciate this time my company, Meta, sponsors me for this event.
I spent 3 days at the conference. The first day is more about academic work form professors and PhD students. The other two days are more about best practice in the industry world. The topics among AI, Distributed System, Advanced Tooling, Product Introduction and Personal Experiences.
I find conferences really help people to think out of the box. And I hope I joined Meta earlier and understand our own implementations so that I could have a deeper understanding of the problem and better compare solutions.
Research Topics:
AI content generation:
Generally, it gets more and more popular. However, the idea is already there. The topic is focused on the gaming content generation. Talk about the best practice of that. Talks about setup rules and how can make it generate more “comfortable” contexts for better game experience
Breakbeats analysis
A topic talks about AI works in breakbeats(pieces background music used in songs). How the company could use that to better understand songs evolution and help them to get song’s patent fees
Textual analyze on code
A talk about how people better understand what code from language-side of view is. And how this kind of thinking could provide people with a new way to understand our code pattern and code practice. Eg:
Trace is one pattern for people to understand the story, it is also a way we used to find bugs.
When we do bottom-up implementation, we always start from the central piece in the structure. This is reasonable from the language side
Product Introduction
Google’s universal IPC language: FIDL in fuchsia
A system very like thrift, but it is used in a deeper system, for IPC. Also doing the binding, so it will support different languages. And easy to extend to other languages.
Workflow:
A new way for making a durable system
A practice that brings “transaction” in DB into service codes, easier for people to write concurrent code. It will be easier to do error handling.
The idea is to make actions more atomic. And for the actions that could change the states, make them idempotent
Filibuster:
An End to End test tool, more focus on testing error cases in end-to-end test cases, to help understand big system’s errors
Talk about in a big system, how end to end testing is needed especially for error cases. Those tests need to cover multiple failing situations.
RLBox:
A way to run native code in sandbox to avoid native code’s memory issues
Talk about how native code is unsafe, and how many safe languages are built on them
Try to run all the native code in the sandbox to make it safer.
It is already used in firefox
A way of coding, so it will be very easy and clear to add to exist system
Akita software:
A product/practice to add API listeners for developers to understand old systems.
eBPF mode to add listeners to the APIs in the old system. And their product provides a solution to better understand the system based on those listeners.
Shopify solution for flash sale:
solution in both LB layer and service layer to ensure capacity, starvation and fairness problem
Very good topic to talk about flash sale, talked very detailed about how they choose from different solutions.
Runtime Checker:
Java best practice to enhance system in observation, watchdogs and in prod regression tests
Very good talk about why we need runtime checker in system, and why sometimes even current wide used system could have bugs
Talk about fix them in three topics
In observation, the idea is that using some static code analysis tools, to find those errors handled locally but not reported to any response.
In watchdog, talks about better catch errors, so the errors could report those error into some monitoring system
In tests, define some testing rules to describe the system, and when running the test to verify whether those rules are still valid. (like behaviors testing along the workflows)
WNFS:
a guide on creating a local-based encrypted file system
Talks about how to create a good file system, including files and directory. And how to keep the history and how to merge versions
Also, talks on how to encrypt those files
Coding Best Practice:
Polymorphism:
different language, different solutions for better polymorphism: Overloading, Override, Interface, Parametric, Structure, Union type, ADT, Multiple dispatching…
Large amount user could search for their health code status
Change user’s health code to unhealthy code based on their time and location if they contact with other unhealthy code
Requirement
User’s health code change could happen with certain delay. ( If I contact with a unhealthy code people, I might get update to unhealthy code 1 day after )
Assume user need to show their health code to get into some place, which means that user’s phone is always opening and providing their data to us. ( There will not be if people go to one place and we cannot get the data )
Design
The whole system could be separate into 2(3) parts:
Part that support user check their own code. Heavy read.
Part that collect user’s time and location data. Heavy write.
When time goes by, one certain range of data collected in part-2 will be complete. And according to requirement-2, we can assume all the data in that time range is in the database and no more to write in. That time we could get the data out and put into an offline batch process, where we will find all the unhealthy code’s movement and get all contacted health code user. We could let automatic process or human administrate to determine the final code change.
Summary
The design helps to separate the read, write, and online, offline, so each part could optimize itself.
Recently, my new project is pushed to production. You might think that I would be exciting about it. Because it is the second project that I designed & developed & tested, and it finally pass the review from senior engineers and goes to the production. But I am very exhausted these days about this project. I have to admit I was 100% exiting about it at the beginning, but at the end now, I was not excited at all. So what happened? At beginning, when I get the project, I would say it is a very good project for me. The requirement is mostly focused on system designing, because the old version is already solve the core algorithm, but people are not appreciated for the stability. What I need to enhance is the system. It needs a more stable and scalable structure, so it will fits more about the production situation. So I was very exiting about this “message queue - worker” pattern implement. I draw the design diagram, I worked on the API document, I create the schema of mongo database collections. And because I learnt a lot from the bugs from the first project I worked. So I think I covered all the corner case and it should be perfect. So after the development finished, I start to push it to production. I was behaved like the project owner, I talked with customer success team to understand their needs and requirements and of course the timeline. Then I talked with cloud ops about the deployment requirements. After that I believed the project is good to go. So I push it to the channel to go production. But the things turned ugly at this point. As I described before, this is not a brand new deployment, but a upgrade, though it is a overall upgrade. But apparently our senior engineers are not agree with that. After a week of silence, My new design is heavily challenged. At first I feel it’s OK, because people’s concern is all reasonable. For example, the structure of the message queue. I will say my message queue is well designed, because I carefully review the code from a open source message queue implementation, and I used message queue from my first project, so I was well prepared to explain my design. But their concern is from the other pointer of the view, the reason they wish I change the structure is because they want to apply the auto scaling policy they are using now to fit for my workers. I never thought of that. This indeed is a very good suggestion, so I changed the code and message queue to fit for the standard message queue structure so they could apply the auto scaling policy to it. ( For a little more explain about this, the other service will monitoring the message queue, and if the queue is slow, they will add machine as queue consumer, so a standard queue will help to determine whether the queue is slow or not ) So I took 4 suggestions at this point and go back to enhancement my code. and after two weeks, I push the code again. After that I waited for a week and people agree to proceed to deployment again. And they required a small change at friday afternoon, so I said that I will changed it next Monday. And I did the change next Monday, but cloud team disappear again, I waited two weeks, and they came back requiring stress test and very strict code review. So this project is delayed and delayed, I tried my best to fit for their suggestions but the bar is higher and higher, and I need to wait more on their deployment plan. I was so exhausted on the waiting. And from this point, I was not excited and try to be like the project owner, I went back as a developer, I just tell everyone what I did and waits for requirements from the others.
What I real feels, is that from that point, I changed from a code lover to a code worker. Yes, before I LOVE to code, so I did the job, and I was really happen about the results. But now, I was more and more standardized. I have to say that I learned a lot from the standards and they are very useful. But I lost my love of this project. I appreciate this change made me a better software engineer but I don’t appreciate this project to made me a code-worker. I hope I could get experience and won’t lose my love of coding.
回家我就开始打state of decay游戏，一款病毒丧尸游戏。看着里面的丧尸和注射解药，不知道为什么觉得真的非常的应景。持续打了大概3个小时游戏，当天我强迫自己睡一个午觉。脱衣服的时候惊觉自己的手臂抬起来已经很痛。之后晚上开始看4小时的超长无聊电影。 第二天正常在家工作，干到下午5点，手上活干完了，想着稍微休息一下准备看一下明天的安排，突然发现自己眼皮非常酸，就开始躺床上休息。此时大概是疫苗打后30小时，然后开始发冷，我记得当时我看室外温度是21摄氏，室内我一直开空调是76华氏度也就是25摄氏。但是我必须要盖我冬天盖的毛毯，不然就会觉得冷。睡了一会到7点，丧失皮肤的冷热感觉，开始口干，但是手臂酸痛开始缓解。然后我就开始了喝水上厕所的循环，我的饮水量大概是一般时候的一倍。期间我量过自己口腔体温，36.6并没有发烧，但是窝在被窝里开始发汗，我换了薄被子仍然发汗，总体感觉就像感冒。 然后就是今天，起床时候已经没有发冷和疲惫状态，手臂酸痛更轻了，洗了一个澡后正常上班。到目前没有什么异样。
I bought a NAS, I am going to use it to do backups, in the meantime, enhance the whole networks.
I have an VPN server, I hope I could use that to access home service from outside, instead of using DDNS or static ip.
I wish to open some service from my home networks, eg: vs code server, jupyter notebook server, so that I only need chrome for mobile devices. And for some small projects like this blog project, I could put the code in the server and use one machine to maintain the blog.
Design
Old Design and Problem
The old design is showed above, Solution is to used a server to run ubuntu as a middle router, and it needs to provide:
connection with the outside internet
connection with the VPN server
provide dhcp assignment for inner-network
Speed become the biggest problem of this. I found that OpenVPN has a speed limit here, normally 100Mbps is the limits. That is unacceptable. And the VPN server is not physically near my home, there already lag, but I thought I could do compromise about the lag. But not the speed, never compromise the speed.
New Design and solution
The new design is showed above: The solution is to separate the whole inner-network into three parts:
modem will assign/lease the ips of 192.168.1.xxx, this will be used for most the device to connect to the internet, it is a full speed network.
raspberry pi will assign/lease the ips of 192.168.2.xxx, which will be used to auto connect to VPN, it is a limited speed network.
vpn server will assign/lease ips of 10.8.0.xxx. Inner service will be open to this network.
So, if want to get full speed, choose modem network, if want to get vpn, choose raspberry network. Raspberry pi will forwarding ports to VPN networks, so when connection from VPN, they could access inner-network service. And even though we expose this service, they are still inside the VPN, so it is very safe.
ubuntu useful tools
As we mentioned before, the core of this design is in the raspberry pi. There are many manipulate on the ubuntu server in raspberry. I read a lot to learn this. So I have this part as a reference for next steps, here we will talk about some tools for future use.
crontab
cronis a very common ubuntu command for timer behavior. It provide a trigger, so at a certain time, it could trigger a action. We introduce it here because it has a config option @reboot, which could be used to add some command at the time of the server boot up, this is much easy to create a ubuntu service to do the same thing.
install：
1
apt-get install cron
run this
1
crontab -e
add this line in the file：
1
@reboot Your-Command
screen
This is a very common ubuntu tool for running a long-live service, it will help you to keep a service alive, even when you close the terminal. Here are some common command：
openvpn client: This command will help you to start the client, still you will need to provide configs and password before execution.
1
sudo openvpn --config xx.ovpn --askpass pass.file
ubuntu netplan && network
If you used ubuntu desktop and change to ubuntu server, you will find this netplan is introduced after 16.04(?) and replace of the network. Still you could change back to edit /etc/networks/interfaces with some extra settings, but we will talk about netplan here. netplan provides some configs in different files.
1 2
00-installer-config.yaml 50-cloud-init.yaml
And you can use one command to create combined rules for test run
1
sudo netplan --debug generate
At the end you could apply the rules
1
sudo netplan apply
And each file is a yaml file，there is a example for a static ip connection setting：
The first one means that, when all the connection in the post-routing status, set them to handled by network interface tun0. So that all the network will pass to tun0 to handle. The second one means in the phrase of pre-routing, forwarding all the request for port 21 to port 21 in 192.168.1.21.
And if you want to delete the rule above when you make some mistakes, simply change A to D here
I followed this blog to start, and did some adjustments on my own： Base
Change ip
Set up dhcp server
Config ip transfer
Setting NAT routing
Most of the steps are the works for me in 1-3 steps, and I will need to do some more the step 4.
How I setup my NAT routing rules
Let’s talk about network interfaces first I have three network interfaces, they are represent to three networks, two of them set in netplan, and tun0 is created automatically when vpn connection set up:
eth0 is used for 192.168.1.xxx
tun0 is the vpn, used for 10.8.0.xxx
eth1 is used for 192.168.2.xxx
And my post-routing is very easy, like this:
1 2 3 4 5
Chain POSTROUTING (policy ACCEPT 14213 packets, 1060K bytes) pkts bytes target prot opt in out source destination 39 2702 MASQUERADE all -- * eth0 0.0.0.0/0 192.168.1.0/24 2 96 MASQUERADE all -- * eth1 0.0.0.0/0 192.168.2.0/24 4101 2089K MASQUERADE all -- * tun0 192.168.2.0/24 0.0.0.0/0
Some explains:
For all connection target 192.168.1.xxx they will be change to eth0 to handle
For all connection target 192.168.2.xxx they will be change to eth1 to handle
The rest will go to tun0 First two will used in port forwarding next, the last one is used for inner network connection
Then you could use command like this sudo iptables -t nat -D PREROUTING -p tcp --dport 21 -j DNAT --to-destination 192.168.1.2:21 to do port forwarding. Here I forwarding connection to port 21 to 192.168.1.2
How to save all routing settings persistently
First of all, beside tun0, all the rest settings could saved as the blog talked before using iptables-persistent. Base the only problem is, when updating the rules, I have to reinstall iptables-persistent and saved again, otherwise it wont saved, I dont know whether this is a bug. However, this cannot work for rules related to tun0, because tun0 only exist when the VPN connection is established. So if you saved the operation above, when you do the restarting, it will auto change will is tun0 to eth0.
1 2 3 4 5
Chain POSTROUTING (policy ACCEPT 14213 packets, 1060K bytes) pkts bytes target prot opt in out source destination 39 2702 MASQUERADE all -- * eth0 0.0.0.0/0 192.168.1.0/24 2 96 MASQUERADE all -- * eth1 0.0.0.0/0 192.168.2.0/24 4101 2089K MASQUERADE all -- * eth0 192.168.2.0/24 0.0.0.0/0
So what we will do is to used auto start to start VPN client first, then add the NAT routing rule. The way to do it is to use crontab -e to add a command to do execution.
When you try to setup network, you might lose the connection to the server if you make mistakes. Before this I didnt bought the hdmi cable for raspberry pi, so I have to reset it several times to get connection back. So I highly recommend you put a plan B here when network connection broken, use a hdmi to change the mistake back. Also, when you do some changes for the whole networks, you might still have a wifi connection from the top level. So if connection is error, you still have a connection to the internet from wifi, you can still do some search on the problem. And it will be easy to set it back when you finished.
This post will provide many basic concepts for entry level blog user. I believed that before you thinking about how to implement something, you have to understand what you are building and why you needs to do that. Back to the days when I know nothing about web or blog, many tutorials I was followed will simple tell you to do step one, step two, but I dont know anything about what this step for. So this time, I will be first talk about these basics, and try to make them unrelated with the languages. If you already have good understanding of networking, you should skip this post.
Website
My way to understand the website is that the website is the resources on the server. When user using the browser, they open the resources which come from the remote server, like images, texts. Then the browser will download the resources and do some manage and they will show you the web page. Image you are using MS Word, you open that, then you choose a file to open. The browser is just like MS Word, the address you put in browser is the address your file you opened. Only difference is that you opened a remote file and you cannot edit but read-only.
Server
Server is just a remote computer. You are not using your usb-mouse or usb-keyboard to interact with them, instead you are using internet signals. In server, you need to put on your web pages and open them to everyone, so people could get this by internet. Start from here, your website is open to visit. Every server in internet will have a ipv4 string, mostly. And if you rent server from others, they will provide that. You can think of that is the unique name of your server, using which everyone will find this particular server, and using which you could login this server. When you buy/rent a server, I strongly recommend you to read this to do basic settings. Some server providers are AWS from amazon, azure from MS. And Linode, Vultr. For me, I use digital ocean. I would recommend you AWS, because it is the most used provider. And digital ocean is good to entry levels, they have many posts and instructions for documents. Here I will provide my refer code for digital ocean.
Domain && DNS
Your server’s ipv4 is just like a phone number, apparently no one could remember all the phone numbers, so no one will remember your ipv4, everyone will remember something like google.com / facebook.com. So the domain is used to replace that.
The biggest domain provider is goddady. and I am using google domain now, they both are good domain providers.
There are some domain provided free, like Freenom have some domains are free to use, they are good for you to play around. After this you will need to link your domain with you server, this could be done with DNS configuration. In some cases, domain provider may not provide DNS for you, you may need to find yourself a free DNS provider. It is not difficult, you just need to set something called A record. Most DNS provider will have documents to tell you how to do that. A record is just like put your number into the phone book.
SSL certificate
Most website nowadays will provide something called certificate, that is ssl certificate, it is used to encrypt you webpage. Image online shopping, when you request your website from your server, is just like shopping online, the delivery guy is delivery your resources to you. If you dont have a ssl certificate, it just like no box of the package, so the delivery guy will know what it is right away. There is a very easy way to get ssl and totally free, that is let’s encrypt. They have a very easy way for you to do it, I strongly recommend you include that in your website.
Non-server implement
Some instruction you might read tells you that you dont need a remote server nor a domain. So why is that? Actually, these two things is provided by others. Let’s take a look at this very good example github page. Every github page will provide you a domain like xxx.github.io. This domain is actually a sub-domain for github.io( all domain comes from back to front, normally you see them in the format of xxx.xxx, here you see aaa.bbb.ccc, you will know that is sub-domain for bbb.ccc). However, the subdomain actually does the same thing as normal domain, they all replace the complex ipv4. In the similar way github also provide the server, because you actually post your page data to repo in github, so the git is just like a server, it contains the resources, it could be managed by git, it could also managed by web. So no server implement is also have server and domain.
Frontend & Backend
Here we use a not strict definition here. As we talked before, open a website is: find the server, request the resources, get the resource and show them to users. In this process, backend is referring to when people find the server, backend will be that one to response to provide resources from server to people. And frontend is actually the resources, when the browser gets the resources, frontend code tells the browser how to manage the resources and show the web page to the user.
Image Hosting Service
You put your resources in server, but that does not mean you have to get all your recourses from one server. FOr example for the pictures, you can post them on your server, but also you could use a image hosting service. So the images won’t takes all your storage.
Google Analytics & Other third party
Google Analytics are very easy to use tool for the website owner to do virtual analyses to learn about the usage of their website. Those service, including salesforce, are main source for data used for marketing campaign. They will gather data from people who visiting the website, normally they will get location info, time info and click info from the user. Web analytics are very big area, I like to recommend you to install it and try, it is totally free. And about installment of these kind of third party service is really easy. First you need to register in their website, and get a id to represent the data you will get from the page. Then you will ingest a code, which they will provided to you and the id together into your webpage, that’s all. Then you just need to go back to their website to see the results and beautify charts.
RSS
RSS is a kind old stuff, but I still think it is cool. You could get a abstract from every posts in web in certain rule and put them all into one file. That file will increase when you write new posts. So other people by listen to the changes of this file to know that you provided new posts. They could use RSS reader to read the abstract of your posts and even open your post to read more. Creating RSS file is also very easy, you just need to make that file to public.
Google Analytics是一种很方便的数据统计可视化软件来帮助网页管理者查看网页的使用情况。这种服务还有saleforce等目前是市场运动很主要的工具和数据来源。他们通过收集访问网页的人的信息（位置信息，时间，点击内容等等）来分析网站。比如如何调整用户界面等等。web analytic是一个很大很新的一个市场，这里就不过多展开了，只是建议大家可以装一下体验一下。 像这类的第三方服务安装有一种共同的方式。首先用户在对方平台注册，获得一个代表将来从我方网站上获取数据的标识。然后将网站上提供你给你的代码和这串标识合起来，插入到前端页面之中。这样就完成了，简单吗？之后就只需要在对方网站查看数据的结果就行了。
After nearly 2 years at this field, now I know how to speak before anything to be done. Yes, you need to ask the million dollar question: could you clarify your requirements? Before I build this Blog, I asked myself what is the requirements for this blog? Here are some of mine requirements:
First of all, easy to build, easy to maintain. The reason I built this blog is to write stuff, not fixing the blog all the time. I don’t want to waste all my time on blog system, I want to focus on the writing, Ok, I didn’t write much
It should be easy to write, so I will be willing to write the blogs. If the blog is hard to use, than I may hardly use the blog.
It has to support my personal domain: senlyu.com . Yes, this domain is so damn cool, why not using it?
It need to be flexible, right now it is a blog, but one day if I find something interesting, I dont want to open another system to support that feature. The potential features I am referring to here is the API system.
Choices:
Ok, let have a look for some choices:
WordPress
It has been a while for me to look back to wordpress. The first blog I build is using wordpress. I think I followed the digital ocean instruction, step by step, and build a stable wordpress, even though I dont know much about apache and front end style. Still I get a nice webpage and using that to get this job. WordPress is good at writing blog, even you know nothing about blog system, you can write in docs, you can write WordPress. But the building process is a pain in the ass, I don’t want to maintain a apache system for this. And it it not flexible for me. Although, I could choose online version from wordpress.com. But if I want to use my own domain, they gonna charge me for that. SO, bye bye WordPress.
Ghost
My first project from my job is to build a system, which is using ghost. So ghost is a old friend to me. Ghost one of the biggest blog project is JS community. Easy to write like WordPress, very easy to post a post. And I am very familiar with the system. The problem I am facing is that the installment. At first I tried digitalocean auto install, this install process will provide you installment including database, ssl cert and all other necessaries. However, the validation of ssl cert always blocked me to install. Because when I link the machine with the domain, it will take some time for dns to setup the link, but the install process will be right away to install everything. So I go and check the documents they have, I found it is outdated. So I think this is a dead end. If the installment is ready this hard, no way it will be a easy maintain. By the way, there is a way I could do using ghost, just like I did in my first project, is to use something like this. This is very cool, although the biggest problem is that this could only run in demo/development mode. it is not a production solution. Still this is enough for most of the blog use case. But I wish I could do it better. So see you Ghost.
hexo + express
So I came to the last choice, md2html blog system + backend support. I use express all the time in work, I am already very familiar with it, ( by saying familiar I mean I worked on more than 1 year on 3 projects all using express, and read all the documents on web, especially on deployment recommendation, and even a little bit source code). So why not express? And hexo, is a md2html blog system. It is not easy to write like above two options, but still good for me.
So
My final choice will be hexo + express. That will be the most easy way for me to start this project.
我工作的第一个项目就是一个含有ghost系统的项目。所以我非常熟ghost。 Ghost是js上面最大的做的最好的blog项目了。写起来跟wordpress一样方便。 但是我真正遇到的问题就是安装，我试了用digitalocean的自动安装过程，它会自动帮你装好数据库，ssl等所有用到的工具。但是验证ssl的过程老是卡住，因为连接云主机和domain的dns设置不是马上起效的。然而我翻阅了他们的文档，我发现极度过时。这可是大忌，装的时候都是问题，维护怎么办？果断跑路。 额外介绍一种我们工作中用的ghost的方式。是类似如下的ghost as middleware. 但这个是demo/develop版本的ghost，ghost的production版本比较复杂也不适合这个模式。如果使用production版本的ghost，就很难和express相结合。no no
