Categories
Uncategorized

Whenever you feel down

Just remember, you felt it before, and before you feel like you will feel down forever. But between then and now, there are moment that you feel happy. So even if you feel like you will fell down forever, you’ll probably feel happy again, just like the last time you feel down.

Categories
Uncategorized

What I learn from trying to make a stock picker

First of all, it is ridiculously hard. If you think it’s not hard, you either, missed some variables, or ridiculously lucky. The more accurate my simulation gets, the harder it is to find a good strategy. A strategy that works before, may not works now.

Stocks prices are very hard to predict. Finding good company is not very hard. Finding good company at a good price is. The better the company perform, the more people buys it, and therefore the more expensive it gets. The reverse may also be true, but at if a company is performing poorly, it may get even poorer or worst, get delisted, so even if the stock is cheap, its not necessarily a good buy.

Lots of people says that we should look at the intrinsic value of a company to know the fair price. The problem is, what is its intrinsic value? Is it the value if the company is liquidated, or “book value” as it was called? Consider this thought experiment: A company that have a book value of 1 billion USD. However, it does not have any dividend, and it is guaranteed that it wont have any dividend, and it will never initiate stock buyback forever. It is run well enough that it will never go bankrupt or shrink in business, let say in terms of income, but not well enough that it will never grow. What would be the marketcap for this company? Considering that it will never have any dividend, what is the upside to the stockholder? Considering that it will never grow and will never initiate any stock buyback? Why would this company’s stock price go higher? If you know that price would never go up, why would you buy it? Considering no one will buy it, everyone would want to sell it, and therefore the price would fall. But if the underlying company will never go bankrupt and will always remain the same, why would the price go lower? You can say that it is a good hedge, but why choose this over bond which will yield a tiny, but some return? This company is, in some essence, what bitcoin is in terms of intrinsic value.

So, we have establish something. Dividend is an easy… intrinsic value of a company (assuming the company will never get delisted and run well enough). A second factor is harder. A stock’s business must grow. But why must it grow you may ask? Well, if the business grew, the stock price would also grow. That however, is where the complication comes. A stock price is determined by the market. The market is also, trying to make money, and therefore, try to predict the intrinsic value of a stock in the future. Let say the market assume that the sales of the company is the measure of the fitness of the business and therefore it’s instrinsic value, and the price follows the business’s intrinsic value. The market would then want a business that will increase in intrinsic value, as it assume that the price will increase. However, the act of buying the business increase the price of the business, possibly above it’s current intrinsic value. The market would keep buying the business, as long as it’s current price is lower than the future expected intrinsic value. If the current price is the same as the future expected intrinsic value, and the future price of the stock matches the future expected intrinsic value, would the price of the stock increase anymore? If not, why would the market buy the business? If it will, and the market does buy the business, the price of the stock would increase and therefore why would the price of the stock increase?

The back and forth create a balancing act which means, a company’s marketcap usually does not match a company’s intrinsic value. It’s usually match what the expected future intrinsic value of the stock. This causes another question. How far in the future?

Here comes another complication. The future can change, a lot. The further the future, the more chance it can change. Additionally, money now, is more valuable than money in the future. Therefore, the future intrinsic value must be higher than some percentage of current price. Because the future is unknown, there is a chance than the company’s future intrinsic value does not match the expected intrinsic value to break even. Therefore, there is only a certain amount of futureness that the market would be willing to predict. The more stable a company is, the more forward looking the intrinsic value is. However, it also means that the current price of the business will get higher.

This is without considering the fact that some of the market, would also try to predict the future price of the business, instead of just the future intrinsic value. As the future price of the business is likely the predicted future intrinsic value of the business, when predicted in the future, this price of the market would still buy the business, in hope to out-predict those who only predict the intrinsic value, leading to a very high valuation.

In conclusion. I got nothing.

Categories
Uncategorized

I need to sleep

Oh my brain, please stop thinking. I need to sleep. Lets think about whatever we want to do tomorrow. No no…. we don’t need to write a blog post. We need sleep.

Do you get me? Let me sleep please…. We will ponder on our life choices tomorrow.

Time goes on whether we like it or not. Time does not slow down or get faster. As we think about time, time moves on. There is nothing we can do about it.

Ah here we go again…

Time is life. Time wasted, is life wasted. Life wasted, opportunity wasted. So much opportunity wasted. Yet, so little time. Wasted opportunity imply the existences of it. At the same time, there is only so much that we can do.

Its not about the opportunity that we choose, it’s about the opportunity that we don’t. But choosing something, imply something else that is not choosen. In another word, there is always opportunity not taken, because we took some other opportunity.

There are life not taken, because we choose another life. Is one better that the other? No one really knows. When people are asked about what do they regret, its usually about things they don’t do. At the same time, they probably don’t do it, because they choose to do other things. If they do choose it instead, would they regret not doing the other thing?

Are we bound to regret regardless?

Let me sleep brain…

Categories
Uncategorized

Should I buy a new or used car: A Malaysian perspective

Here in Malaysia, we have 65% excise tax on cars. If you pay RM 500 a month for your car, about RM 200 of it goes straight to the government. Not to Proton not to Perodua or Toyota. Some say that does no apply to local manufacturer, but I’ve yet to find concrete evidence that they are exempted from it.

I’m 26 years old that have no car. Its not that I can’t afford them, I just really hate loan. But its way past due that I take the bullet and accept this constantly depreciating item in my life. Which brings me to my question, new or used car.

First, some background, I’m single, I go to work with a motorcycle. I will most likely not going to use the car to go to work in order to prevent more of my money flowing to the government. So fuel consumption is not an issue. Its most likely going to be used to balik-kampung to Kedah. Once every two months perhaps. Car must have ABS. I had a scare before somewhere near Slim River where the Axia I used got a case of locking brakes. Car must be practical, really, the main purpose of the car is to carry stuff that I can’t carry with my motorcycle box. I have like a big table, that preferably can fit into the car. Folding rear seat is a must. MPV is even nicer. Car must be reliable or do not depreciate much. This means its mileage should not be too high. Or, its so cheap that it can’t go even lower. Car should be locally made (not necessarily designed), because its easy to find parts or repair. This leaves us to Proton or Perodua.

Preferably, I don’t want to have a loan at all. I have some money, but not enough to buy a new car. I thought of getting a dirt cheap Proton Saga to save money. But those generally don’t have ABS. More importantly, the old saga cannot fold down the rear seat, a puzzling design choice. A Viva does not even have ABS option. This leave a Myvi, but I really like the 2011 “lagi best” model, which is still expensive. To my surprise, I might as well get a new one, which is only about 50% more expensive on most case. I thought, cars depreciate really fast, but this is a 8 year old model. Why does it still cost this much? I’ve looked at carlist.my, mudah.my, facebook, all more or less the same. Which makes me think, is this the norm? How much do car really depreciate in value?

Looking into it….

Using my programming skills, I’ve scraped a website that list car for sale in Malaysia, which will be kept unnamed, for legal reason. Unlike mudah.my, this site make sure that they don’t list deposit price, or sambung-bayar price, only sale price, although generally its not OTR. This make it easy to scrape, no need to clean data. I only scrape Proton and Perodua models, remove those whose mileage is more than 250 000 km, and only consider automatic transmission. Also, they can’t be more than 15 years old, and I excluded listing from ‘dealer’, as I simply don’t trust them. Majority of the seller here are agent or private seller. And because I don’t want a really old model, I’ve specified the model name explicitly.

as_pd = pd.DataFrame.from_records(flattened)
as_pd['mileage'] = np.ceil(as_pd['mileage'].astype('float') / 5000) * 5000
as_pd['price'] = as_pd['price'].astype('int')
as_pd['year'] = as_pd['year'].astype('int')
as_pd['age'] = as_pd['year'] * -1 + 2019
as_pd['seller_type'] = as_pd['profile_type'].apply(translate_profile_type)

cond = (as_pd['model'].isin(['Alza', 'Myvi','Axia', 'Viva', 'Bezza', 'Exora', 'Saga', 'Persona', 'Iriz']))
cond = cond & (as_pd['mileage'] < 250000)
cond = cond & (as_pd['age'] < 16)
cond = cond & (as_pd['transmission'] == 'Automatic')
cond = cond & (as_pd['seller_type'] != 'dealer')
perodua_pd = as_pd[cond & (as_pd['make'] == 'Perodua')]
proton_pd = as_pd[cond & (as_pd['make'] == 'Proton')]
all_pd = as_pd[cond]

So what kind of population are we looking at here. For giggles, I’ve disabled dealer filter for this occasion:

Hmm, what brand is sold by who?

Fascinating! It seems that although in total, while there are more Protons listed, they are generally sold by dealers. Hmm…. Anyway, lets ignore that because we are not considering dealers. Then the most listed car is without much surprise, the most popular car in Malaysia, Myvi. Most listed by private seller and most listed by agent. The next privately listed car is Proton Saga. Surprisingly the second listed car by agent is actually Alza, not Saga.

Side note, I actually really want an Alza. But their price does not go down in the used market, not much at least. Now, let us plot their listed price over their mileage. This should show us how much value a car lose over the distance it traveled. I’ve splitted the scatter graph between Proton and Perodua, as 17000 points in a single plot is very hard to decipher.

Price over Mileage

Perodua
Proton
Both

Simply said, there is no easy correlation between mileage and price. We do have some pattern however, after 100000 km, the price does not drop much anymore. Depending on car, 60000 – 80000 km is a good target to look far.

There is a strange anomaly at 10000 km, where there is a bunch of cheaper car. I suspect this is some bad data, some seller putting refurbished car or just saying car with less than 50000 km as 10000 km.

There are other patterns also. Viva pretty much don’t change in price. It is cheap in the first place, and does not go much cheaper. For some reason, Iriz increase in price as mileage grow? From the scatter plot, there aren’t many car being sold when their mileage is lower than 40000 km. I suspect the sellers just say they are 10000 km in mileage.

From the scatter plot, there aren’t many car being sold when their mileage is lower than 40000 km. I suspect the sellers just say they are 10000 km in mileage. That said, mileage just does not affect the car price much and it highly vary from seller to seller. Really, what you want to look at (at least what sellers are looking at) is the age of the car.

Price over Age

Perodua
Proton
All

Now we are looking at something. The price of a car is somewhat proportional to its age. There is a lower bound however. No car is priced below RM 10000. Brand-by-brand, Perodua does command a higher price that Proton. In fact, Alza is consistently more expensive than an Exora at age 2+. At 4+ year old Axia is actually more expensive that a similarly aged Saga. At 7 year old Saga and Persona have identical price, AND priced more or less the same as a Viva? In fact, at 8 year old, an Exora is priced similarly to a Myvi.

That said, Persona wise, this is the old persona, not the new one. Iriz still goes up in price a bit. Bezza also have a bit of a price hike at age 3. Is a newly released model more expensive when it first lunch? A new version does bump up the price a bit. For example, 7 year old Myvi is significantly more expensive than an 8 year old Myvi, due to the 2nd gen model. Axia is significantly more expensive than Viva no matter how you look at it.

A few things to keep in mind, these are all listed price. It is most likely not OTR price. Add 3k to 5k for that. It is possible that the buyer can haggle for the price, which lowers them, but I’m no good at that. Simply said, the actual buying price could be higher for me.

The price depreciation does slow down a bit. There seems to be a ulower bound of RM 10000, or RM 20000 for MPV where their price will not go lower. Or maybe the model is just not old enough for the price to go lower.

It seems that a good estimate of how a price of a car drop is to say that it have a half life depending on the model. Meaning the price will halve over some period of time. That said, its not like 2 or 3 years. Perodua seems to have a half life of 6 or 7 years. Saga and Persona have a half life of 4 to 5 years.

Much of this is perhaps just market sentiment. Perhaps. I know I don’t want a second hand Persona, because I’ve used one. It had a broken fuel gauge, then the immobilizer immobilize it, something wrong with the computer, completely replaced the dashboard, still broken fuel gauge. But that is the old Persona. Iriz have a favorable review, and the new Persona is based of Iriz. Saga is so simple, not much can go wrong. Exora on the other hand, have a turbo. And a bunch of canggih stuff. See how the price drops.

Conclusion

If you wanna buy a second hand car, buy a Proton. If you wanna buy a new car, Perodua is a better choice. Me? Old saga cannot fold rear seat down. New saga are 25k+, listed price, not OTR. OTR is probably 30k. I might as well buy a new one at 35k.

Myvi is a better choice. Lots of people selling them. A ‘lagi best’ EZi model of mileage less than 80000 can be found for 22k, which OTR is probably 25k. And its rear seat can be folded down FLAT. However, it does not really have a bonnet, which means cargo length is limited. Bezza is too new, and its rear seat is vertical. Alza however, not only have a flat folding rear seat, it have TWO rows of flat folding seat! Making it a van, which is awesome! But, I can’t find a good one below 30k, list price… When a new one is 55k, its hard to justify buying a 9 year old car for slightly more than half price.

So, did I make a decision? Nope. Why? Simply said… I’m cheap.

Categories
Uncategorized

The case of a Key-Value pattern.

I’m quite conservative about how my application store its data. I prefer SQL over NoSQL. I prefer it’s security of the dynamic of NoSQL, which you would eventually probably use an ORM which have schema anyway. Relational databases have been used for decades, so it must work right? And really, most of us will probably never going to need the performance of a NoSQL database. That said, there are good uses for things like Redis, or other form of Key-Value store, notably for simple stuff like configurations. But for real work, for me SQL is the right way to go.

But what if you don’t know the structure of the data of the application? What if there are no structure? What if there are indeed patterns, but there are many special cases where in the end, the structure does not matter much.

For those who are not technically familiar with relational database, in relational database, there are things called ‘Table’ and that is where we store data. For example, if I have many car, I will put create a ‘Car’ table, and in the car table, I will declare the car’s properties such as the height and weight. But all car must have the same attribute. Sometimes, if some car don’t have a certain attribute, we just don’t fill the attribute for that car. That works. Sometimes, instead of cars, we have to represent vehicles, some can be car some can be bicycle. And there are many ways to represent this data, but the gist of it is this, all things of the same type generally have the same properties.

But what if out of a hundred car, two car have a special properties which no other car have. This is what I call special cases and I hate special cases. It makes the code ‘unclean’ and kinda makes me ‘itchy’. Sometimes, these two car have special calculation that need special treatment. Generally we put the name of the car in the configuration file and then special-handle them when they occur somewhere in our code.

In my latest project, I have a bunch of these special cases where this data is the same as these data, or these data is equal to these data multiplied by 100. But only these particular data. In this case, the number of ‘car’ is fixed and there are predefined number of them. But some of it have special characteristic which is not the same as other stuff. On thing the got in common is that they represent values over time. Or in another word, I was to keep record of what are the weight of this care at this particular time. And some car does not have weight.

The reason for this is that previously the business flow of the company is to use Excel sheet with various formula. And some cell is this cell multiplied by these cell. Using Excel is fine by itself, the problem is, half of the time, there are no obvious pattern to derive from. It is very hard to design the sql table for this case.

The original code of the project uses a key-value system where each value is mapped to a key. There can be multiple value per key which also have timestamp of when that value is active. And then in clientside there are list of formula of how to calculate certain values which is calculated from other values. Previously these ‘formula’ pattern is used in two place. In a place where the used periodically key in values, and a background process where it will fetch data from another service, and transform them using these formula. There are also other background process which run some calculation on these values to get some other values such as average, but those are hardcoded.

At first, I modified the key-value formula system so that the formula is stored in the database. Then it is calculated on clientside when needed as most of the calculation is for view purpose. Then came the graph, and some of the graph’s value is calculated from other value. That means if I were to calculate it on clientside, the browser would probably hang, and calculating it on demand would take too much time. So, I make a background task, kinda to calculate these value to store then for graphic purpose. And then some of the value require average of other values, which mean it needs access to multiple values over multiple time. And then I make it work. I make sure to sort the calculation order topologically for correctness, in case a calculation depends on another key which is also calculated. I make it as fast as I could by caching the data used by calculation before sending it to database, which significantly improve its calculation speed as it does not have to hit the database. I make the formula work while having partial set of data, without hitting the database at all, allowing the clientside to ask the server to quickly calculate the resulting value which allow the user to see the changes before it is saved.

Managing this large number of keys is a huge hassle, so I make a template system for the configurations so that I don’t have to repeat repeated patterns. To save space, value segment next to each other that have the same value is combined, effectively compressing the database. When the formula of a value is the key of another value, basically it just link to the other value, and storing the duplicated value is a waste of space and compute resource. So the storage layer is modified a bit so that it just point to the actual value. Linking is a great feature as it allows us to define mapping for a page and then link them to actual data later one. Which is really important as the data may came from multiple other page arbitrarily.

In another word I’ve made it in a way that any time-series number data can be relatively easily represented and linked to each other. Now here comes the big question, Why should I use it? Everyday I ask myself this question, and everyday it always comes to “How else to do this?”. You see, when you made such complex things you kinda ‘own it’. And you want everything to use it. Much like when I made a monadic LoadableResource pattern. But is it really needed? A question I ask myself everyday. My co-worker said that other developers don’t understand it. But ‘other developers’ is the kind of developer which needs to be explained the different between an array of string and an array of dictionary. What does they understand?

Some of the page now does have some pattern. But to me that sounds like it will need to create two more additional table, making sure the query is correct and making sure the calculation is correct. No to mention some data comes from other page. And the structure of the other page may not be the same. The list of car on the other page is not the same as the list of car in this page. And how are you going to map from one page to another? By creating a configuration file. I know, lets make sure all configuration file is stored in the same place and for mapping between two page, have the same structure. That sounds a lot like the whole value mapping system before, except there will be multiple places that do the same thing.

For me the biggest downside is managing the mapping names. There area a lot of them. Up to 6000 right now. But they are defined in multiple file, each file separated by its domain. And there are the template system. Its just that no one is using them except me. Even so, it still feels a bit cumbersome for me. If it gets even more complex, it might as well be its own programming language. Additionally, if it makes so much sense, why haven’t I heard about such pattern before? Is it just very domain specific, or I am just missing the term to google it? Who knows.

Categories
Politics

Mahadhir is Harapan’s Prime Minister

Less than 8 hours ago, the opposition coalition here in Malaysia just announced that Tun M is their candidate for their prime minister. And that my friend, is the very reason why BN is still in power, and probably will continue so. Its not about Tun M becoming the candidate, no, its the very fact that the opposition for some weird reason came to that conclusion.

Tun M, their old arch enemy is apparently their president. The prime minister who exceeded his maximum term during his time, is now apparently should get another term? At the age of 93.. something? I’m not saying that he can’t do it, I’m saying that the opposition is very, very disappointing for even considering it.

I have no doubt that Tun M have huge influence and follower. I have no doubt that he is capable of doing so too. He leaving BN is a huge blow to BN. Arguably, he is the single most influential political figure in Malaysia right now. My mother cried when Tun M resigned, which he did live on television, probably so that he can’t back out, and its too late for people to object. Yes, people asked him to take back his resignation, on the spot, live. This is with him having roughly 20 year of being a Prime Minister, which is twice of what is allowed. I kid you not, this is the kind of person Tun M is. That is why Tun M is a huge deal. But as a Prime Minister, again? No. Its not right. Its not proper. The new PM for the new term should be a new guy. Its not about what he can do, its about what the opposition can’t do. Don’t they have anyone, anyone at all to be the candidate? Can you entrust the future of the country to such people, who seems to only and ONLY want to win the election, above all else?

And when you think about it, not much. The opposition does not have a candidate that truly stands out. I disliked PAS for breaking their pact with PKR and DAP. But I’m glad they did. Because if they do, they will have to deal with this kind of political BU!!$#!T.

That said, not everyone, probably not most people, will share my opinion. Having Tun M as a candidate will gain support from the rural area, which traditionally is BN’s stronghold, not to mention, he IS BN’s kryptonite. These people are generally of the elderly category which will probably see this as a positive thing. The younger generations however may share my opinion, but may also condone such ridiculousness in exchange for BN’s dismissal. It make some sense.

BN for all its corruption, at least have some professionalism or rather… authenticity.. or rather… I can’t find the term to describe them. They are bad, but at least they are properly clever. They are the ‘stable’ choice. The status quo. That said, I’m trying my best to find someone else to vote against them. And my option is shrinking. Lets just hope PAS will not do anything stupid before the election.

Categories
Uncategorized

Jobless and graduating

So I finally quit. It was an okay quit, I guess. A but abrupt, yes, I admit. But it had to be done. And right now, I’m jobless and I probably got enough money to survive until the end of next month.

The thing is, it turns out, finding a job is hard work. Finding a suitable job, that it. I tried ServiceRocket, did not even get an interview. I registered to a hiring firm, got an interview to a small local company. Did not pass. Apparently I’m “not ready yet technically”. “not ready yet”, I would agree, “technically”… I’m not exactly sure. But I couldn’t say that is wrong either. I don’t really have much to say on things that matter.

If I learn anything on my strange internship, its that technology does not matter. What matter is what effect do the technology bring to the table… the meeting table… with the client in front of you. Sadly my previous job do not offer much in that sense. Something that I’m paying right now.

But hey, I got to start somewhere right? Which is also not that easy for me. In my CV I have nearly 4 years of experience. But without much impact, those experience are poor experience. Starting somewhere, would mean entry level, and how am I going to get an entry level job with 4 years in my resume? The interviewer would be like “Are you hiding something?”, which I kinda am. Those are remote jobs, they are not very…. impactful.

Anyway, after that interview I tuned up my resume as high as I can. It feels a lot more impressive now. Anytime I feel down, I look at it, and it makes me feels a tiny bit better. I gone through my backup from the year 2014 and see what I did with Coglab.biz. Put everything that I think ‘impactful’ in my CV. It feels a bit too ‘low-level’, but I don’t have much ‘high-level’ thing to be proud about. Hopefully that is enough. I sent it to several bigger companies today. But today is Saturday, they’ll probably process those on Monday, which brings me to the next agenda… Convocation.

Ahhhh great. I’m going to meet old friends and when they ask what am I up to right now, I’m going to have to say “Unemployed”. And then they’ll say “Seriously?” And them I’m going to say “Yup!”. And then a series of uncomfortable conversation. My Convocation is this Monday by the way. Which also brings other complications to my resume. I hate this. Its like my birthday and my friends wedding. It only reminds me of what I did not do, and its too late to do it.

As the job hunting continues, I have several leads right now. The resumes I sent to those companies, may take more than one week to process. Abidzar said that he have some friends that want to make a new team, a not-web-development team, and I could join it if I could hold on until next month or so. And he sent my resume for a contract job with Petronas, and another C++ job, I dont really know where. I got an interview from a recruiter for a job in Japan. Its a bit too far, but if I did not take this up, I’ll regret myself. There was another recruiter from Singapore. I turn that one down before, because I don’t want a job outside KL, well at that time at least. I turn down many recruiters before. Man, they can be quite handy now.

Anyway, wish me luck for my convocation.

Categories
Uncategorized

28th September 2016

I’m bored. I wanna quit my job and work on something else.

The guy at one of the talk I’ve been said to the audience, “What do you wanna do?”

I honestly have no idea. Today I got an email to an invitation to Asia’s Rice Bowl awards. As an audience, not a participant. I don’t know how did I get the invitation. Zu did not get it. Abi Dzar did not get it. I know I won’t go there. I got nothing to show for it. If there is anything I learn from Superb and Symbiosis, among these guys, I’m no one.

Heck, who am I? Some might say a really good coder. But in these few week, one thing I’ve learn is that, even in that aspect, I’m not really that good. I just have an early head start. And if I stay and my current job, I’m gonna be left behind. It really is quite sad. Without my skill, who am I?

I could not even do ImbasPay well. Not even Bahasapp. I could make excuses here and there, but the fact is, I really don’t know how to manage a project. Need to go and work somewhere else.

Thinking about Superb and Symbiosis, I really have a talent of getting myself into places where I probably should not go. Likely due to my presentation skill. A quirks I get back from secondary school. You see, sometimes when I do some public speaking, it can work really well, and people for some reason have the impression that I could be a champion or something. Unfortunately I have a very poor social skill which makes me awkward as hell. And in business, leadership and I guess life, social skill rules. If I could code my way through it, I would. I’m kinda am.

Oh well, what am I blabbering about again? Yea, changing job. The thing is, I need to complete ImbasPay and then the IIUM research thing and then I can safely change job. Which is a lot of work. Sometimes I ask myself, do I regret making choices that lead me to today. Well, then I respond, “If I keep thinking of these regret, I won’t be able to move”. So many regret. But it does not mean much today. I really need to learn how to say “No”.

What do you wanna do?

Categories
Programming Projects

List of Quirks when using Spring Framework and Kotlin

I’m not used to the JVM environment. Usually I use NodeJS, Ruby with Rails or PHP with Laravel. But after awhile, I find these dynamic programming environment quite problematic, especially on a large project. Rails is simply too magical, it is dark magic. Sure, it have a nice syntax, but debugging them is quite a hell and no type safety means you really need to unit test your software. PHP and Laravel is not that bad actually, but when the things that makes it nice came from statically typed language, like type hints and IoC container, why not just use a true statically typed language? NodeJS have quite a lot of things. You can have type safety with typescript. You probably can get an IoC container… if you want to use them. But the whole NodeJS environment is way too immature and changes too quickly making it feels quite brittle. Update a package and the whole thing could collapse. Not too mention typescript support really depends on the library maintainer. Its the same case for flowtype too. Except I had worse luck with it.

So, when you want to use a statically typed programming language to make a web application, two things came to mind: C# and Java. Other languages are too not-mainstream enough. With C#, you have .NET MVC and with Java, you have Spring, Jersey and Play framework. The open source version of .NET which is .NET Core is simply too immature. So whats left is Java. Java is quite tedious, but luckily, we have Kotlin, which is another programming language running on top of the JVM with very high interoperability with Java. In the choice of framework, Play framework utilize Akka which is way too different than conventional application. Jersey is actually quite nice, but it does not have a built in integration with other things. Spring is the most famous of them all, so its not that hard to choose, its just that it could be overwhelming.

First of all, what is Spring framework? You need to get this right real quick, because the libraries built around it is quite a lot. At its core, the Spring framework by itself is only an IoC container. Spring MVC is a web framework that have the Model-View-Controller architecture that we want. Spring Data is an umbrella project for data access related things and among those is Spring Data JPA which provides integration with JPA (Java Persistence API) which is a java specification for ORM (Object Relational Model). Spring Data JPA utilizes Hibernate which is a JPA implementation.

Now do you see what I mean by overwhelming? All of these libraries are configured by declaring a Configurer bean. A bean is basically a fancy way of saying class/object with specific structure. You can also configure through XML, but that is old school technique which result is a huge mess of XML. The new way is too declare a Configurer class and configure it with code. In general you’ll probably use a ConfigurerAdapter instead, because Configurer is an interface and you don’t want to configure everything. But still, that is a lot of configuration. Which is why they created Spring Boot, another project which is basically a collection of configurations of these libraries. Spring Boot will detect if you include these library and will automatically apply a default reasonable configuration. ‘reasonable’ does not mean convenient, or in some sense even reasonable. I guess a better word is ‘compatible’ or ‘safe’. Still, it is too invaluable that you probably should google ‘Spring Boot’ instead of ‘Spring Framework’. But what if you want to adjust some parameters of some library? Well, it could be that Spring Boot have already enabled the use of configuration files. Which by default is usually application.properties. But you can also use YAML. I’m not exactly sure the file name for that. But what are the settings? Here comes the problem with Spring framework in general: the only documentation is its user guide/reference and those references are very long. That, or you go through the source code of Spring Boot or the respective library, which are Java code, which are very long.

Once you get through those long and tedious readings, assuming nothing goes wrong, you get the ability to:

  • Create an MVC controller by annotating arbitrary class, with the base path and so on.
  • Declare a a bean which is an interface that extends CrudRepository<Model, Int> where Model is your arbitrary POJO (Plain Old Java Object) that have certain annotations that declare it as a model. And inject it to your arbitrary other bean (like your controller), gaining a free implementation of your interface that interface with your database.
  • Include liquibase in your gradle, then just declare your migration in resources/db.changelog/db.changelog-master.yaml.
  • Ability to map http body to arbitrary object, effectively parsing JSON request to a POJO, with a single annotation.
  • Return a POJO from your controller method, and it will automatically convert it to JSON.
  • Annotate your POJO with java.validation annotations and annotate your POJO controller parameters with @Valid and it will throw error if the JSON send does not validate.
  • Ability to annotate your bean method with a security expression, and it will throw error if it does not pass.
  • Java’s plain simple and conventional generic and inheritance model.
  • Probably other things too.

If that sounds quite magical, well it is. Maybe too magical, that if somethings goes wrong, you don’t know what happen. Good thing its all mainly Java code, so you got nice IDE support for debugging. But its a whole lot of Java code. A whole lot.

Of course, its not all good. Some of it is quite bad actually. The template is JSP which I’m not a big fan off. You can’t easily rollback your database. I use it mainly for a rest controller. Unfortunately in that regard, Jersey brings more things out of the box. When a validation error happen, it does not report it as a nice JSON. Jersey does that nicely. When a resource is not found and so it fails to bind the resource as parameter, it simply pass an empty resource. Jersey return a nice 404. And of course, when something goes wrong, I’ll take a day or two to find the problem. So here is a list of problem/quicks I found so far. I’ll probably update it as I found more issues.

It is asking for a password, some csrf validation error? And CORS issues.

By default, the spring boot configured spring security will ask for password, which is randomly generated. CSRF protection is enabled, not really useful for REST. Also, CORS will need to be allowed. So I used these config to overcome those issues.


import org.springframework.context.annotation.Bean
import org.springframework.context.annotation.Configuration
import org.springframework.security.config.annotation.web.builders.HttpSecurity
import org.springframework.security.config.annotation.web.configuration.EnableWebSecurity
import org.springframework.security.config.annotation.web.configuration.WebSecurityConfigurerAdapter
import org.springframework.web.cors.CorsConfiguration
import org.springframework.web.cors.CorsConfigurationSource
import org.springframework.web.cors.UrlBasedCorsConfigurationSource
import javax.inject.Named

/**
 * Created by amirul on 31/05/2017.
 */
@Configuration
@EnableWebSecurity
class WebSecurityConfig: WebSecurityConfigurerAdapter() {
	override fun configure(http: HttpSecurity) {
		http
			.cors().and()
			.csrf().disable()
			.authorizeRequests()
			.anyRequest().permitAll().and()
	}

	@Bean
	fun corsConfigurationSource(): CorsConfigurationSource {
		val configuration = CorsConfiguration();
		configuration.allowedOrigins = listOf("*");
		configuration.allowedMethods = listOf("GET","POST" ,"PATCH");
		configuration.allowedHeaders = listOf("*");
		val source = UrlBasedCorsConfigurationSource();
		source.registerCorsConfiguration("/**", configuration);
		return source;
	}
}

Model Validation does not work with kotlin.

If you use a data class with the constructor arguments, make sure you annotate the field instead of the getter/setter with something like @field:NotBlank.

In a one to many column relationship, on save, the child’s foreign key is not set, and I’ve specify it in database as non-null, so it will crash.

In a a JoinColumn annotation, make sure you specify nullable = false, like this @JoinColumn(name= "parent_id", nullable = false)

In a one to many relationship, if I remove the child item from the array from the parent object and save the parent, the removed child is not removed in the database.

In the @OneToMany annotation, make sure to set orphanRemoval to true.

I’m getting `org.hibernate.HibernateException: A collection with cascade=”all-delete-orphan” was no longer referenced by the owning entity instance:`.

Yea… the collection in the entity is pretty special. So you can’t just assign them and hope the complex datamapper technique to work. So don’t replace the collection object itself. Call clear and addAll instead.

I’m getting `org.hibernate.AnnotationException: Collection has neither generic type or OneToMany.targetEntity()`.

Try not using Kotlin’s immutable List. Which is the default list. Use MutableList instead. Yeah, I know Hibernate does not play well with immutable data types at all.

Categories
Projects

Automatic IIUM Schedule Formatter/Maker : End of life announcement

The Automatic IIUM Schedule Formatter/Semi-Automatic IIUM Schedule Maker is probably the most used application that I’ve ever made. That is impressive or disappointing depending on how you look at it. For some reason (probably because there is a problem to be fixed and hence, a demand) quite a lot of people use it. However, because I’m no longer studying in IIUM, I’m no longer one of the people who use it. And because of that, there is effectively no reason for me to support it anymore. Therefore, I’m declaring it life as effectively ending. I make no guarantee that it will stay online.

capture2
Here, we see a very ‘seasonal’ usage pattern.

For your information, the application itself did not earn me any money. It does, get me some nice portfolio. Quite a lot of people asked me “Why don’t you sell it to IIUM?”or “Why don’t you put ads on it?”. Well, the short answer is, “That sounds complicated, I’m too lazy for that.”. The long answer is, I don’t want it to ‘matters’ too much and I don’t want to get some ‘feature request’ from some guy, and I don’t want to get into trouble if someone found a bug. It is really, a ‘pet’ project, and I prefer to keep it that way. Plus, asking money from IIUM for something that should do in the first place does not sounds like a very good idea. I sense a lot of bureaucratic issues on that path.

Because it is running on my private server which also host other applications, like this blog, it does not really have a hosting cost. I kinda ‘piggyback’ it on other application. Plus, it DOES automatically scrape data from IIUM server every Sunday (But it turns out, sometimes IIUM change it last minute… after Sunday). Therefore, I don’t really need to do anything for it to function normally. However, every year, I do need to set which year is available, which involve editing about… 2-4 line of code.

Selection_016
This is its original purpose

As I’m too lazy to put any further significant improvement, I’ve open-sourced it several years ago. It seems that not many people know that you can get the source code and see how it works. Probably because I did not put a “Fork me on Github” banner before, which I just did yesterday. I hope that some other geeks can learn from its source code, and create their own version, or better yet, create a totally new one, as the code is quite ugly. The code originated nearly 5 years ago when I was still in CFS. Back then, the popular ‘schedule maker’ part did not exist, only the ‘schedule formatter’ which works on CFS’s slip exist. It was originally designed to run on Google App Engine as I don’t have money for my own server back then. Because of that, the code is pretty much some hacks over some other hacks. Therefore, I recommend that those who want to make a similar system to build one from ground up. You can also, just host it yourself. In fact, when the new I-Maalum system was announced, I was expecting them to also integrate a similar solution, or even just straight up copy-paste it. That did not happen.

So when will it go down? Who knows. The thing is, removing it from the server also take some work, and I’m too lazy for that. I’ll probably just leave it there. If my server crash, I may forget to start the application manually (which happened a few month ago). Once it is up, It does not really need much maintenance, so it will probably stay up for some time. However, for the 2017/2018 session, I’ll need to add that to the server’s config and I can’t guarantee that. Chances are, I’ll forget about it. In fact, I did forget about the 2016/2017 session until someone mentioned it to me. That is assuming no major changes in the IIUM system. If IIUM decided to change the format of their course list page, the scraper will not longer work and it will no longer give you updated course list.

capture
Fork me!

In conclusion, tell your geek friend to make a similar system. I’m graduating, I don’t even know when is the prereg, start of semester and such. I thank you all for your kind support and encouragement throughout these years, I appreciate it. It is truly a satisfying feeling to know that your work helps a lot of people, even if it does not get you money, and the code is very ugly. Also, if you need to contact me, email me directly at asdacap@gmail.com. Please do not PM me through facebook, as facebook have the ‘message request’ filter. Sorry, if I did not reply to your facebook message if you messaged me a few month ago because I did not realize it, and replying now would be very awkward. That is all from me. Goodbye and Assalamualaikum.

ps: Try reading this article again, but everytime you find the word ‘lazy’, change it to ‘busy’. Its interesting how changing the meaning of a single word can change a lot of perspective. Also its not far from the truth too.