Categories
Code Misc

Deep vs Wide problems and Icebergs

I recently had to explain why a task was taking longer than expected, and I needed a place to vocalise my thoughts, so here we go. In my experience generally there are two kinds of problems which tasks can be put into(not some tasks can be both). Wide problem being something is not necessarily difficult to implement but it’s changes require changes all over the code bases, an example being a blocking mechanism. Deep problems are problems which will be complex but in a very limited domain. An example might be adding a new feature which is distinct and does not rely on too much other code.

Image of wide vs deep problems

Often wide problems can become what I like to call icebergs where tasks seem simple however their edge case causes the task to drag on. In addition they can upset other existing functionality.

Image of a iceberg problem.

A big problem with wide problems is they can be very demoralising for teams, as their wins are often small and finding and fixing edge cases can be difficult and time consuming. In addition teams will often just want these tasks to finish because they are dragging which leads to tech debt to accumulate.

Deep problems on the other hand are often easier to deal with as new functionality is often well defined, however making sure tasks don’t grow wider is key to making sure deep problems don’t become icebergs. Deep problems sometimes can be difficult if for example the feature being added requires lots of domain knowledge, or knowledge of legacy systems. In order to avoid this make sure other members of the team can do knowledge sharing about features, so other teams are not in the dark.

Categories
Code Help

Binary Search explained

I believe in order to understand something you need to be able to explain it to somebody else.

In that light here is me helping you helping me. I will try
and do this for other data structures and algorithms.

Binary search is a search method for searching in a ordered list. Order in this sense meaning the list has a hierarchy that might be purely numeric/temporal but it also could be lexicographical like alphabetical ordering.

How does it it work?

Here is the pseudocode

left = 0
right = length of array - 1

while left <= right
   mid = rounded middle of (left + right) / 2
   if mid = target return mid
   if mid > target right = mid -1
   if mid < target left = mid +1

It seems simple enough right? Another way to think of this is we are taking the list checking the middle if the middle is larger we look at the two sub arrays and do the same depending if the middle is larger or smaller.

Worked Example

Let’s say we are looking for 19

Let’s now add the pointers

Now let’s add the mid in python //2 will round up

Since 11 is less than 19 we now make left 17 since we do mid + 1

Now let’s add mid

Since we have not reached the target we can exit.

Time/Space complexity

The time complexity for this problem is O(log(n)) and the space complexity is O(1), The reason the time complexity is Log is because we are taking half the array each time. The space complexity is constant, because we are not requiring any additional data to be stored beyond the pointers to left right middle.

FAQ’s

Does the array need to be ordered to work? Yes, though the algorthm will “run” it won’t be possible to get the right answer because a random ordering will mean it would not be able to arrive at the answer.

If the array is unordered wound’t it make sense to sort and search or search the list directly in a brute force manner? Excellent question If we use a sorting method like merge sort(my favourite due to it’s consistent performance) that would mean O(nlog(n) to sort + O(log(n) in computer science we would resolve this to O(nlog(n) this is greater than O(n) which is the brute force way of searching an array, meaning sorting and searching would be slower.

Categories
Uncategorized

How I would Learn Programming from Scratch

I have been programming earnestly for about 8 years and professionally for 5 years. I have experience in front-end development, back-end development, and data science. My path is fairly traditional, considering I initially learned by myself and then continued my education as part of my undergraduate studies and during my placement year.

  1. Start with Python. Regardless of whether you plan on learning front-end, back-end, or data science, Python will help you grasp the fundamentals without getting bogged down in syntax, which is something I struggled with initially. While syntax is important, it can be a barrier for many people. I started with Java, which wasn’t a horrible idea, but I spent a lot of time learning higher-level concepts like OOP that were only useful much later on.
  2. Start creating things as soon as possible. There can be a temptation to watch YouTube videos, follow guides, and copy and paste the results, but this often gives a false sense of learning. When it came to tasks outside the domain of these guides, I struggled. Instead, look up how to do the specific subtasks needed but not the entire project. Learning a language is similar: knowing words and syntax is important, but knowing how they fit together leads to fluency. Example projects to start with include a diary application, a to-do list, and Hangman. All of these are CLI utilities, so you won’t get bogged down in display methods.
  3. Learn how to read documentation. It might sound trivial, but being able to read and fully understand documentation without examples is a key skill that will be invaluable in the future.
  4. Start reviewing other code and look at how projects are structured, particularly on GitHub.
  5. Experiment with other areas of programming. For example, if you are interested in making websites, try learning JavaScript or TypeScript. If you are interested in making applications, try using Tkinter. If you are into making games, consider looking at Godot.

It’s important to have fun while learning. If you don’t enjoy the learning process, you might find programming hard to enjoy. Learning new skills is a massive part of programming, and it’s a constant journey of learning from others.

Categories
Code Misc Personal projects

Making a cloud Native webcrawler in Go

Image of nodes connected in a graph. Showing how pages related to each other

Over the past few weeks I have been making a webcralwer. I wanted to do it as way to get better at Go with useful for learnings for Graph databases as well as being fun. The project made use of cloud native items such as AWS SQS, DynamoDB and optionally Neptune which could be swapped out for Neo4j.

What is a webcrawler?

A webcrawler or webspider is a program which visits a website, and fetches all of the links on that site and then visits them. This is how sites like Google/Bing/DuckDuckGo get the pages to populate when searching.


Architecture

Image of AWS services. SQS is drawn connected to the Webcrawler as well. There is also a connection to Neptune and AWS DynamoDB

SQS queue

The primary role of the SQS queue was to allow as a store of links to explore. It also if there is an issue processing the link we have a dead letter queue automatically which allows for a high level observability.
It also allows for the program to better scale horizontally. By Allowing multiple nodes to pick up the work.

DynamoDB

This acted as the long term storage of the data. Initially I tried modelling the links between sites inside DynamoDB however this did not work well with the access patterns due to 1MB limit on items in DynamoDB. DynamoDB works excellently as a NoSQL serverless solution for Key value look ups

Neptune/Neo4j

I also used a graph database to store the relationships between pages. This proved much easier than expected. Due to openCypher being a really beginner friendly tool very similar to SQL but allowing for very complex relationships to be models easily. I will be using graph databases in the future for other projects as well.


Take aways

For me it was a fun project which I learnt a lot about some of the major difficulties which Search engines encounter. Such as pages not being formatted correctly, or pages linking to subpages endlessly. In the future I want to build a searching mechanism on top of this to act as a crude full search engine.

For anyone curious here is link to the Github

Categories
Misc Personal projects Tech

Deployment options for Rails

Over the past year, I have needed to deploy Ruby on Rails applications to the cloud, and I have tried a few methods.

  • Capistrano
    This cool system and is fairly interesting, but not the most modern technique it creates files called cap files which then can be used to deploy the app over SSH. However the main issue being scaling, since it only deploys to one location it means auto scaling becomes difficult.
  • Heroku
    Heroku is a PaaS system, meaning you don’t have to deal with server configuration, the application runs inside a specially configured docker container. In terms of ease of use Heroku scores very highly, Heroku makes it very easy to deploy an Application to production. However it being a platform you pay a premium for the ease of use.
  • Docker
    This methods seems the most hands on but it grants the most options, and probably the most modern.
    My recommendation is to create a deployment docker file, which can build the file from a Git repository. Then its your choice if you use something like kubernetes, or Docker Swarm to create other containers with the app running.
  • Dokku
    Dokku is an interesting project. The program uses docker containers in the backend and configures them. So in a way you can host your own PaaS system, it requires some configuration, however It can lead to a very effective PaaS System.

In the end, the type of deployment you do depends on the requirements of the project.

Categories
Code News Personal projects

New Code blog

Collie.codes is the new blog for Projects I am working on.

 

I have been keeping this on the back-burner for a while now, but I have finally gone live with a blog to keep people up to date with all of the projects which I am currently working on and all of the projects I have finished. In addition I made a Scrum board for my self, which works for each project listed on the website .

Which is very handy for keeping up to date with projects and seeing what needs to be done at a moments notice, I took inspiration from the TV show Silicon Valley, for the layout of this particular scrum board. It also in the backend as the ability to add keys to highlight particular tasks. EG Networking related things might be Green, but I didn’t end up adding to the final build since I am the only one using it so the tasks are pretty clear to me what are they fall under.

 

Categories
Uncategorized

UAV Object recognition

Overview

Recently I have been working/designing a UAV system, to be able to detect people, then from that be able to relay this to a ground station. I like it call the this system UAVS (Unmanned aerial vehicle Sighting)

Reason

On large ships, when a person is overboard (MOB), there is very little that can be done, apart from release a Lifebuoy, however often in large waves, it is very easy to lose sight of the MOB. So using a UAS system it means that you can get an aerial view, hopefully allowing for the MOB to be found and rescued.

Technical Overview

I have planned to use a single Raspberry Pi to run both the servos, to allow for the 2 freedoms of movement, also allowing for the camera to seek and find objects on the ground/sea.

The Raspberry Pi is running a software called YOLO which is an object detection and recognition system so far has been very reliable about identifying objects. However since the Raspberry Pi CPU power is very limited it means that the there is a significant drop in frames during the recognition process which could be problematic. However the solution is simply to use a more powerful device. 

side view
Top side view
Data flow schema

Overview of Systems

  • Plane: Bixler 2
  • Camera Simple HD Usb camera (However Thermal optics, would be ideal for night time operations)
  • AI recognition system: Tensorflow
  • Video transmitter: Fat Shark 600mW 5.8 GHz A/V Transmitter
  • Control system: Futaba
  • Battery: 2x 1600 mAh 3s LiPo (1x for Motor Power and Another for the Raspberry Pi)

Possible problems

Though this solves the problem of sighting, it still needs to be autonomous, right now it needs a person to operate the UAS, and observe sightings in the video link. So would require training and special equipment which is not practical for many instances.

Another possible problem is that since the drone is very light in order to maximise flight time. It means that in high winds like during a storm it will be blown very quickly away from the ship. To combat this it could make since to use a Quadcopter in order to have more control during storms. However during light winds it makes since to use a light plane. Since it allows for a greater distance to be covered.

Future

I will try and update this Blog with all the progress from this project, so expect updates shortly.

Categories
Code Misc Personal projects

Dev-ops on an iPad

The surprising power of an iPad

Recently I have been testing my skills/patience by trying to make and develop websites using an iPad (Pro 10.5 with smart Keyboard).

SSH

SSH initially seemed like a big hurdle since many web based SSH Clients are Java based Applets. However using an IOS app called Coda which comes with a SFTP/FTP client and SSH this was no longer and problem, however there are is probably a limitation with the usage of PEM, however I have limited usage with PEM, so your mileage may vary.

MYSQL

This is probably the weakest part of the whole experience, there are many apps however which can help with database management, in addition using phpMyAdmin might be the easiest and cheapest for most people.

Web development

A major problem which I found is that the iPad cannot run any server-side code (php, Ruby, Node.JS). So It was limited to just HTML and JS. However the workaround is to use a VPS, to use as a development server, and run and modify the code remotely, this is not ideal however at a pinch it is a viable alternative, especially if you create a disk image of a pre configured server, then all you have to do is start a VPS with the disk image and transfer the file you need to work on.

In iPad review

Though it is possible to deploy and run websites just from an iPad, it can be very frustrating especially when moving files around, however IOS 11 has helped the file experience on iOS. The experience is still sub par compared to a desktop. However it is nice to know it is possible, however I would only consider doing this with either an external keyboard or a smart keyboard since using the onscreen keyboard seriously limits usable space.

Categories
Code

My New/Old Keyboard

IBM Model M

I am a big fan of mechanical Keyboards, having used and owned one since 2013 (yea not that long). However I have decided to upgrade my keyboard to the legendary IBM Model M, heralded due to its Buckling spring design.

The IBM model M is like nothing I have ever used before. It has an unparalleled tactility and provides a physical and audible clunk each time a key is pressed creating one smooth and fast typing experience. The Cherry MX switches which I’ve used before provided an ok feeling and noise, however the underlying plastic below the keys lead to inferior sound quality when compared to the Model M.

When I Initially purchased my IBM model M I realise that I’d accidentally purchased a terminal edition keyboard, meaning it had a different layout and a different connector, using a rj45 better known as ethernet port. However it wasn’t a traditional ethernet port, requiring a special converter so I had to order a part from America to make it work. But I’m so glad I did.

It has led to an increase in my typing speed and overall is a much smoother typing experience. If you do a lot of typing like I do it’s probably worth getting a mechanical keyboard.  Many people including me speculate that the spring nature found in mechanical keyboards leads to an increase in typing speed and accuracy because your fingers build better muscle memory between keys and can be brought back up faster with physical and audible feedback. Whereas in a rubber dome key switch the key needs to be bottomed out to create a contact, leading to an increase in the time between keys pressed. Thus in a mechanical keyboard it only needs to be pressed around halfway for the key to make contact and the machine to register the key pressed, leading to increasing speed of typing.

 

A demonstration of me typing jargon into the keyboard

 

https://youtu.be/hXl1a4RK_hA

 

Categories
Code Personal projects

Why I moved my hosting from shared to VPS?

My situation

I was a tenant on a larger server. I was suffering a lot of outages which for my US readers was an issue. In addition the package I was on, limited my access to the server, so I could not add php code(Server side programming language) which limited the usability of the domain.

The VPS hosting provider

I had a few choices before I made the move do I do a windows environment, or Linux. I was on a windows provider and I did not see a reason to justify the cost of windows server license. Also I did not really see any benefits vs Ubuntu server LTS(which is what I am using now). The VPS host, I had used both Linode and DigitalOcean before I was more impressed by both. I chose DigitalOcean since I preferred its choice in servers, and configurations. Both offer SSD servers at reasonable prices. They can also allow me to easily add more servers if I need them in the future.

Worth it?

Hell yes!, I have full control over the server, so if I want to turn one into a game server at any time I can. I can easily run all the applications I want(Email murmur). It makes me much more flexible.

 

Is it for you?

Hard to say. If you are familiar with Linux and servers then probably, if not you will get very confused since the learning curve for server is sharp. I would recommend getting a raspberry pi loading Ubuntu server and seeing if you can run WordPress on it before you start paying for a VPS. In addition hold onto the old hosting provider for emails since setting up emails on a server can be a massive issue. Since there are many pitfalls like ip blocking which you can run into.

In the end it all down to personal preference.