Wherever you go, there you are.

Linux router conntrack settings

0

Linux firewall? Conntrack? NAT? Connection issues?

This is one of those "I'll never remember this again if I don't write it down" types of posts. So entirely for my own purposes, basically :)

I have a linux box acting as a firewall/NAT device for my local network. Among other things, I'm using conntrack modules for NAT connection tracking to handle proper NAT port forwards and also for firewall rules to filter active connections properly.

A few issues cropped up in weird places like Netflix, YouTube, VEVO, and other streaming services where streams would die for no apparent reason. I always just chalked this up to ephemeral internet issues and did not investigate it deeply as it was not common enough to be more than a minor, infrequent, and random inconvenience. However, a consistent, reproducible, and most irritatingly CONSTANT problem trying to watch twitch.tv streams finally got me looking into this in detail.

Now, I don't know the details of WebRTC, HLS, RTP, and all the other protocols under the hood for video streaming tunnels through HTTP. What I do know (now) is that the default timeouts in conntrack in 3.x kernels seem to be too aggressive (at least for my internet connection), causing conntrack to often drop the TCP connection tracking for computers using these streaming protocols.

The result? Random drop-outs and network connectivity problems in HTTP-based video streaming.

The fix ends up being stupid simple. I just doubled (or sometimes tripled/quadrupled) the TCP connection timeouts for conntrack and, at least so far, streaming stability has improved dramatically (and twitch.tv actually works). The new timeouts are still short enough that for my limited network size I'm in no danger of running out of conntrack entries in any reasonable timeframe.

So, to the end of my /etc/sysctl.conf file, I simply added these timeouts (Ubuntu 14 LTS system, btw):

net.netfilter.nf_conntrack_icmp_timeout = 60
net.netfilter.nf_conntrack_tcp_timeout_close = 60
net.netfilter.nf_conntrack_tcp_timeout_close_wait = 120
net.netfilter.nf_conntrack_tcp_timeout_established = 86400
net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 240
net.netfilter.nf_conntrack_tcp_timeout_last_ack = 120
net.netfilter.nf_conntrack_tcp_timeout_syn_recv = 120
net.netfilter.nf_conntrack_tcp_timeout_syn_sent = 240
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 240
net.netfilter.nf_conntrack_tcp_timeout_unacknowledged = 300
net.netfilter.nf_conntrack_udp_timeout = 60
net.netfilter.nf_conntrack_udp_timeout_stream = 240

Oh, and while I'm on the topic... I also updated my conntrack modules to handle a few of those irritating protocols. These work much more cleanly now with the proper conntrack handling in place (especially PPTP). To my /etc/modules I have added:

nf_conntrack_proto_gre
nf_conntrack_pptp
nf_conntrack_sip
nf_conntrack_h323
nf_conntrack_ftp
nf_nat_proto_gre
nf_nat_pptp
nf_nat_sip
nf_nat_h323
nf_nat_ftp

EVE ruminations

2

It's been a while since I've posted about EVE Online... or, let's be honest, rambled about it. I felt like posting a new ramble because, well... mostly because I like the sound of my own voice. But also because CCP has been through a lot of ups and downs and their future, and the future of EVE, was looking pretty shaky for a while. Recently, though, it feels as if they have finally found the right track again.

I've always found CCP and EVE to be a fascinating example of game development done... maybe not "right", but at least "well". Since I've been getting hyped about Star Citizen recently, and since it is still so very far away, I thought it might be nice to remind people that there is an excellent sci-fi/spaceship MMO out there that you can play right now :) Not only is it out there, but that it seems (once again) to be getting better every day.

(random self-serving note... if anyone reads this post and is inspired to try EVE, please be a pal and use my referral link)

I want this post to focus on where EVE seems to be going and why I'm excited about it. But it's hard to do that without taking a glance at where EVE has been.

Whereupon I offer a biased and unabashedly opinion-laced view of EVE's history...

CCP -- the creators of EVE -- are an interesting example in game development and the videogame business. They are an Icelandic company with a true "garage startup" genesis story that rose to success based almost solely on the singular strength of their idea; to make a sci-fi spaceship MMO with lasting consequences and an intrinsic PvP focus where everyone is playing in the same (and only) universe/shard. That game -- EVE Online -- launched in 2003 and has been growing ever since.

Starting in about 2011 it became undeniably apparent that the growth and success of EVE may have had as much to do with luck and the power of its niche idea than with any particular genius over at CCP. Floundering and confused choices made by CCP with regard to EVE, combined with spectacular failures in other projects, suggested that many of things CCP did right over the years might, in fact, have been accidental rather than intentional.

Powerful as the niche idea was, in 2011 EVE was already 8 years old. An idea can only take you so far. There was growing concern at this point that CCP might no longer be able to replicate the successful choices that had brought the game as far as it had come, leaving the future health of EVE in doubt. Some of these great chioces, in no particular order: self-published, a downloadable game client, no "box fee", a warm embrace of 3rd party applications and openly available game data, the introduction of PLEX (a free-to-play option that dampens gold farming while avoiding pay-to-win), complete graphic engine overhauls, the CSM, hiring an actual economist to help deal with the player-driven market... the list of intriguing business decisions alone goes on. Sure, some of this stuff seems common-place now, but at the time these things were introduced to EVE they were rarely industry-standard and often bordered on revolutionary. But quietly so.

I'm not going to discuss the things that happened leading into 2011 that made it apparent to players that CCP might not actually know what they were doing. If you are not familiar with EVE you don't care, and if you are familiar with EVE then you already know about this stuff. I will, however, mention in passing that at least one event led to CCP being the only game developer I know with an Internal Affairs division.

What it all led to was a fairly simple conclusion; CCP believed they knew better than the players what EVE was and should be, and always seemed a little bit surprised when there was any kind of player backlash. The issue, however, is that many EVE players seemed to think this was a NEW problem at CCP. They looked back at previous successful choices as compared to some of the more recent disasters and concluded that something at CCP had changed. But it hadn't.

The reality is that most of the previous successful choices made by CCP were made under the same belief that they knew better than the players. They just got lucky with those choices for a variety of reasons, so things turned out OK. There is one classic example that highlights this: wormholes.

In 2009, EVE got one of the most significant and expansive universe updates it's ever seen, leading to a whole new dedicated style of small-group gameplay that revitalized the ability for the "little guys" to make a place for themselves in EVE. It was widely applauded and led to a whole new style of emergent gameplay enjoyed by many, injecting life into the game. But pretty much everything people love most about wormholes? It was a mistake.

Wormholes were essentially intended to be dangerous PvE content for groups. They were effectively a first attempt at what was later re-introduced in the form of incursions in 2011. Nobody was supposed to setup permanent residency and actually LIVE in wormhole space. The fact that you could anchor a starbase in one and create a home for a small band of misfits was entirely accidental and a mere oversight on the part of the devs; they simply forgot to turn off the flag allowing for anchorable structures. And yet claiming wormhole space was exactly what players did, in droves. It offered a welcome outlet to groups interested in PvP and self-reliance but who were simply not large enough or didn't care enough to challenge the sovereignty and political machinations of null-sec.

That is just one instance, but really you can look at the majority of their choices and find similar themes. Titans, capital ships in general, jump drives, "gun mining", moon goo, original faction warfare... the list of poorly implemented game mechanics that were badly abused by players in unintentional ways goes WAY back. That's not even including the poor business choices. Just because most of these things turned out OK prior to 2011 didn't change the basic fact that CCP always assumed they knew better than players and always reacted to player backlash with surprise and a bit of resentment.

Oh sure, they would sometimes get around to fixing things that the players had proven beyond a shadow of doubt were completely broken or imbalanced, but it was always with a tinge of resentment from the developers, and typically MONTHS too late to prevent the exploitation of these broken features from severely impacting all aspects of the game. Further, by the time CCP would wake up to and think about addressing some of these mistakes, often so much time had gone by that the players now saw the broken mechanic as part of the status quo and resisted any attempt to improve or tweak things for the better. This was not helped by the fact that CCP seemed unusually reticient to ever admit that the first pass had BEEN a mistake. By 2011 however, CCP had made a series of bad choices, none of which could be viably salvaged in any reasonable way. In short, their luck had run out.

UphEVEal (yes I know that's not how you spell it)

The realization that they had been lucky and not genius must have come as quite a shock to CCP. Sure, they had always given lip-service to the idea that they embraced player feedback, but reality proved different. They ignored all of the most requested player features, left broken mechanics in place for far too long, introduced new bizarre mechanics from way out of left field, and generally operated in a way that showed their true colors: "CCP knows best". Only by 2011 it was finally obvious to them that if they stayed that course, EVE was going to die (for real this time!)

Watching CCP struggle for the last three years to internalize and act upon this realization has been interesting, to say the least. Again I won't bore with details here; either you follow EVE and have already witnessed the earthquake that is shaking up CCP culture, or you don't care. I'm just going to talk about the result, since the result makes me excited for the future of EVE; enough so that I actually bothered to write this blog post :)

It seems ridiculously simple, but everything that was wrong with CCP and all the hope for their future can be summed up with one simple change; their new patch cycle. Until recently, CCP operated on a 6-month patch cycle, attempting to create two "major content" releases each year. These were accompanied by grand ideas, hype, trailer videos, and all the fanfare they could muster. They have recently changed their patch cycle to a 5-week one, aiming to release 10 updates a year.

It might seem like oversimplification to boil it down to this, but all I can say is that the release cycle has an incredibly insidious and far reaching effect on every aspect of development in a company. For a developer, every feature, every interaction with players, every interaction with your managers; all of it is driven in large part by how it works into the release cycle. Likewise management is evaluating every effort and tradeoff against the release cycle as well. In EVE, the 6-month cycle lead to an 11-year problem; no little features, no big features, no iteration... just "medium" features that sounded cool in trailers.

If a feature was too big to fit into a 6-month cycle, it wasn't even attempted. This led to some of the most serious long-term issues with EVE mechanics to date; null-sec sovereignty problems, poor starbase controls, a general feeling of "static-ness", and ignoring many of the most requested player features going back a decade, such as customizable ship appearances. These are things that the developers have been saying they would like to fix for YEARS, yet somehow they never managed to sort out how to start working on those things within their 6-month cycle.

Likewise, if a feature was too small to help hype up the next 6-month release, it seems to have been thought of as "wasted effort" and was never approved for work. Ditto for iterations and tweaks on recently released features from the previous content update. This led to a horrific absence in quality-of-life improvements to essential aspects of the game, such as the user interface. As for player feedback and "mistakes"; if, as a developer, you have to convince your boss to release a patch off-cycle, you are of course going to be resentful to any player that presumes to point out something broken. You might agree it's broken, but you simply don't want to face the hell of trying to get it fixed to production. So you are probably going to argue (or be dealing with your manager arguing) that it isn't "broken enough" to warrant a patch. Which leads to a broken mechanic becoming status quo, because everyone knows it won't be touched until AT LEAST the next 6-month update. And by that point it's easier just not to touch it at all. If you are an EVE player, this should sound very, very familiar.

CCP's new focus on a more frequent, less-hyped release cycle is the closest thing you will ever see to a silver bullet in development and business choices. Of course in isolation it's not enough, but taken as a whole it's proof-positive that CCP has actually had an internal revolution that it can no longer just be "lip-service" to listen to the players. They actually intend to DO it. I'm certain that the further down the ranks of CCP employees you go, the more you find people who have always wanted to do so while also feeling the futile pain of acturally trying to do so. The new release cycle gives these day-to-day developers the tool they need to finally turn the lip-service into reality. And so far, it seems to be working.

In the last year I've seen more quality-of-life UI improvements than at any point the game's history (FYI: I've been playing since 2004). And I don't mean the "let's overhaul everything UI in Trinity and then leave half of it broken for years" type of update. I mean actual, useful, small improvements that benefit everyone. Things that are finally making it into releases because there is no longer a need to tie every release to a marketing push. More importantly, these improvements have been iterated!

The tooltip update not quite what you wanted? No problem, we'll listen to all that player feedback and actually TWEAK IT with improvements 5 weeks later! I can only imagine how empowering it must feel as a developer at CCP today, being able to finally ACT on the things people are complaining about instead of simply being helpless to respond.

But it's not just little things... for the first time in, well, ever, CCP is finally making tangible progress on some of the largest features that, until now, they had only ever talked about. Customizable ship appearances, starbase overhauls, and changes to nullsec sovereignty. All three of these things had previously been projects so large that they were never attempted.

The funny thing is that at least one of these major changes has EVE players freaking out, because they perceive it as a reversion to the old "CCP knows best" mentality that has burned them so badly over the years. This is the power projection changes being introduced in the next patch cycle (Phoebe) to start working on the null-sec sovereignty issues. Superficially, these changes feel very much like the CCP of old; a set of changes that nobody quite wants or understands, fraught with unpredictable consequences which CCP has not fully thought out, all while players point out significant potential flaws on the forums that devs only seem to tentatively take to heart.

But the likeness is, thankfully, only superficial. What these players are not realizing is two important facets; one, that the devs, after literally years of doing nothing -- paralyzed by the requirement to get a nullsec overhaul done right, the first time, in no more than 6 months -- have finally actually made the first tangible progress in trying to change an aspect of the game that is widely regarded as needing improvement. It honestly doesn't matter if they get it right out of the gate. The fact that they feel the freedom to attempt anything at all is a huge improvement over the previous state of affairs.

Secondly, and more importantly, is the fact that CCP has proven over the last year that they are no longer in the "release and ignore" mentality of previous updates. They have repeatedly iterated on new features rapidly under the new patch cycle, and the 5-week timeframe allows them to do so before those changes become an entrenched part of the status quo. It's far easier as a developer to admit that something was not quite right (or even, gasp, a mistake) when you get to fix it almost immediately than it is when your ego is on the line for the next six months.

What this means for Phoebe is simple; I have no doubt the initial release is going to be not quite perfect. I don't expect CCP to get this one any better than any of the other massive, unpredictable changes they've tried to bring to EVE in the past. What I do believe, however, is that they will finally be able to adapt on the fly. When Phoebe goes live they'll be able to watch the impact and iterate quickly to tweak quickly as needed. That isn't something to freak out over, it's something to be excited about.

Cyrptocurrency Revisted

1

I just wanted to make a quick follow-up to my original cyrptocurrency post from a few weeks back. This just has a few follow-up links and additional specifics I've come across since that time.

I forgot to mention last week that if you want to participate on the Litecoin or Bitcoin networks and have your own personally managed wallet address (e.g. not a wallet on an external exhange or site, like Cryptsy), then the simplest way to do that is to grab the offical bitcoin-qt client (for BTC) and the official litecoin-qt client (for LTC). These are p2p clients that enable the underlying networks of their respective cryptocurrencies, but they also allow you to create wallet addresses for receiving deposits and making transactions.

If you are particularly concerned about your wallet's security (or are just a very paranoid person :), consider using Armory for your BTC wallet. It pulls data from a locally running bitcoin-qt client under the hood and leaves all of the p2p and networking details to that software. Armory instead focuses on managing your wallet addresses as securely as possible. It has many advanced features such as options for wallet recovery and truly "offline" wallets that can be stored on a computer not connected to the internet at all. These offline wallets require physical access to the machine in order to sign/verify transactions (which you can then send out using a "read-only" online wallet counterpart).

On the mining side of things, Trade My Bit, the multipool I use for mining, has added several new features. Most notable is an automatic exchange that allows you to opt-in to a centralized and managed sale of the multipool mining currencies. Yes, there's a small fee for it, but it means not having to worry about linking up to Cryptsy (or your preferred exchange) for every single currency or having to directly manage a ton of microtrades if you don't want to be bothered with the minutae. You can just get a regular BTC payout from the shared exchange pool at the end of the day. Definitely a very cool feature and the devs there have been very active in updating the pool in general.

If you are using AMD GPU's for your mining efforts, this guide is a detailed step-by-step process for setting up a headless cgminer on Ubuntu. And this optimization guide will help you get the best possible hashrates out of your Radeons.

Lastly, for the AMD crowd using cgminer you are probably aware that 3.7.2 is the final version that supports GPU mining; all further releases of cgminer are dedicated to bitcoin mining on ASIC's and drop GPU mining support. However, someone has taken up the banner and forked cgminer into sgminer. It provides on-going development of and bugfixes for the GPU scrypt mining capabilities in the 3.7 branch of cgminer.

Introductory guide to cryptocurrency (BitCoin, LiteCoin, etc.)

1

This is a primer on cryptocurrency (BitCoin, LiteCoin, etc.): what it is, how it works, and some specific (though probably ill-informed) recommendations from me if you are interested in mining it.

Crypto-what now?

Cryptocurrency is the name given to any currency based on a couple of key concepts and a common theoretical architecture. The first cryptocurrency was BitCoin, which "set the mould" for what it means to be a cryptocurrency. Another important cryptocurrency is LiteCoin, which has a couple of key differences from BitCoin that will be covered later. In addition, there are now dozens (if not hundreds) of "AltCoins"; that is, other cryptocurrencies in addition to BitCoin and LiteCoin. They have many of the same features and underlying concepts, but have small yet important tweaks or differences that make them suitable for specific niche purposes.

Cryptocurrencies have two central features that make them unique from normal "fiat" currencies:

  • anonymous transactions
  • no central processing or legal authority

Why do I care?

Let's say I have $50 and I want to give you $5. How do I do this? Well, if we can meet in person I can just give you a $5 bill. That's an anonymous transaction to the rest of the world, but it's not anonymous between us. Further, in an online and interconnected world, it's really inconvenient to have to meet in person any time we want to exchange money.

Another option is that I can keep my $50 in a bank account and then tell my bank to send you $5. This means my bank will subtract $5 from my account, then my bank will tell your bank to add $5 to your account. This works because we both trust our banks to perform this transaction (typically backed by government regulations), and likewise the banks trust each other.

This process works really well and is the basis for the modern financial system. However, this process is never anonymous because our banks know a lot of information about each of us. And you will never find a bank that doesn't, because in order to comply with banking regulations imposed by the government in most countries, a bank MUST collect this information about you or else the government won't allow them to be a bank. And if they aren't a bank in the eyes of the government, not only can they go to jail for trying to do "banking" things, but more importantly no other bank will trust them and they won't be allowed to participate in these kinds of transactions, which makes them kind of useless as banks.

On top of this, the transaction itself is logged by the banks and, again due to those government regulations, the processing authorities involved (typically quasi-governmental central bank agencies) can intervene at any point. They could tell our banks to stop the transaction, or reverse it, or even confiscate the transaction (or my entire account) for any number of legal reasons. This legal and regulatory framework and government involvement is, in fact, the very basis of a fiat currency, and is the reason familiar currencies (such as the Canadian dollar or the UK pound) are associated with a specific country. Those currencies are effectively defined by the laws of their relevant governments.

Now, it might sound quite paranoid to worry about a government confiscating my transaction or banks knowing who I am, and I personally believe that, yes, in reality those aren't things most people need to worry about most of the time. Not only that, but some of these things are actually positive attributes that most people WANT in a bank. I LIKE the ability to call my bank and cancel or reverse a transaction in the case of fraud, and I appreciate the fact that if my bank screws it up my government can fine or jail them or order my funds returned to me. These are useful and powerful features of fiat currencies!

But choice is king, and in the modern world there are enough people who ARE worried about the limitations of fiat currencies that there is a demand for a currency that can be used anonymously and without any potential for intervention from any central authority or government. Whether you personally worry about that or not, enough OTHER people were worried about it to figure out a way to do it. Thus, cryptocurrencies and BitCoin were born.

The Secret Sauce of Cryptocurrencies

Cryptocurrencies are based on asymmetric cryptography, hence the name. Asymmetric cryptography is based on decades of research into advanced mathematical ideas that are crazy genius. However, all you need to know are three basic facts:

  • modern crypto is based on having a "key" that is split into two parts: a "private key" and its corresponding "public key"
  • encryption: anyone with your public key can create a message that can only be read using your private key; everyone else just sees it as undecipherable gibberish
  • signing: you can use your private key to "sign" a message (that anyone can read) and anyone with your public key can can verify that only you could have written the message

In this kind of system, "you" are anonymous because no one knows who you are. Instead, "you" are defined by the fact that you have the private key. Assuming you don't share your private key with anyone else, then no one else can pretend to be "you". You don't need to reveal your identity or register with a central authority to prove that a message came from you or to read encrypted messages sent to you. All you have to do is share your public key and then do a good job of keeping your private key secret.

Another key technology underlying cryptocurrencies is peer-to-peer networking (P2P). This is the technology underlying things like BitTorrent. Again it's not necessary to understand the technical details of P2P; all you really need to know is that it enables software on many computers to talk to each other (create a shared network) in a de-centralized fashion. There is no central authority managing the network or controlling who can connect to it, and yet everyone still manages to "get along". As long as your computer can find another P2P client to talk to, you can participate in the P2P network.

Transaction Redux

Given these key bits of technology, let's now examine the case where I have 50 "coins" and I want to give you 5 of these coins. We'll ignore for the moment how I got a balance of 50 coins (or how anyone knows I have this balance, though that will become clear shortly). How can I do this without bank accounts, or even banks, or any central authority enforcing transactions?

In cryptocurrency lingo, I have a "wallet" that contains the 50 coins, and you have a wallet where I want to transfer 5 of those coins. A wallet has an address (a string of letters and digits) and each wallet is associated with a public key. I can prove to other people that I "own" a wallet because I'm the only one who has the matching private key for the wallet. Only I can sign public messages sent from the wallet. Note that in real usage, you'll actually have many wallets. At least one for each cryptocurrency you use, and often several for a single currency that you use for different purposes. This is kind of like having multiple bank accounts.

Now, I can create a message from my wallet and sign it with my wallet's private key that says "Hey, I'm taking 5 coins from this wallet and putting them in your wallet". Now, assuming I can find you to give you this message, you can verify that I am, in fact, the owner of the "from" wallet because I signed it with the private key. Great! I've "given" you 5 of my coins! Of course, at this point only you and I know anything about this transaction. You know those 5 coins now belong in your wallet, but I could easily lie to everyone else and tell them I still have all 50 coins I started with.

The trick here is the P2P network. Instead of only giving YOU the message that I'm taking 5 coins from my wallet and putting it into your wallet, I'll give EVERYONE that message. The entire P2P network will get the message that my wallet is sending your wallet 5 coins. They can all verify that it is a valid message that could only have been sent by someone who has the private key to my wallet. Now the entire network agrees that, in fact, my wallet now only has 45 coins and your wallet received the other 5 coins. Interestingly, you don't even need to get the message for the transaction to have taken place. Everyone else agrees that your wallet now "owns" those 5 coins, even if you don't get the message right away.

Finally, if we track every transaction ever made in this manner from the beginning of the currency, it becomes obvious why the network accepts the fact that I had the 5 coins to send to your wallet in the first place. Somewhere in the history of transactions, my wallet must have received those 5 coins from someone else. A wallet address that has never been seen before must have 0 coins, so the only way my wallet could have 5 coins to send to you is if I can prove via some other verified and confirmed transaction in the P2P network that my wallet received those 5 coins from somewhere else.

This complete record of every coin moving between every wallet is called the "block chain". Each cryptocurrency has its own block chain stretching back to the beginning of the currency. It contains a complete record of every transaction ever made. Counter-intuitively this completely open and transparent system, in which I can see the flow of currency from every wallet to every other wallet, is 100% anonymous. Because each wallet is just a random identifier, and because all I need to prove I "own" a wallet is the private key, there is no way for anyone to know who I am. I can prove I own a wallet and that a transaction is valid without sharing any personal information of any kind. For all I know, every transaction that ever happened in the currency is just one guy sending stuff back and forth between all his millions of wallets.

Further, it's also 100% de-centralized. Because all of these messages happen on a P2P network, there is no central authority that can control the process. As long as I find enough P2P clients to confirm the transaction and the message gets to most of the P2P clients eventually, it's practically impossible to forge or undo.

The Two Big Gotcha's

This de-centralized and anonymous nature does have two significant limitations. One is a simple result of there being no central enforcement body. As a natural result, there is NO WAY to reverse a transaction! Once I create a message that I'm sending coins from my wallet to your wallet and it is confirmed in the block chain, I CANNOT undo that transaction! I no longer own those coins. The only way to "undo" the transaction is if YOUR wallet sent the coins back to me, voluntarily. This means that when doing business with coins you'd better trust the other person or else use a trusted 3rd-party/escrow agent. There is no bank or credit card company to call to reverse a transaction if you are being swindled or defrauded.

The second major issue has to do with the specifics of how all the transaction confirmations are handled and the way the block chain is created over time. I won't cover the very technical details, though I will talk about the block chain more in the mining section coming next. However, the important take-away is that if a single entity gains control over more than 50% of the block mining effort (NOT 50% of the P2P network, but 50% of mining new blocks), they can subvert the entire process! They can lie and cheat and forge information about past and present transactions.

Mining Blocks

If you were paying close attention in the previous section, you might have noticed one missing detail. If every new wallet starts out empty, and if the only way to "own" coins in a wallet is to have a valid transaction showing that the wallet received coins from somewhere else... then where do the coins actually come from?

The answer lies in the specific details of how the block chain, or transaction ledger, is created and updated over time. For a number of technical and practical reasons that I won't get into, the transactions in the P2P network are organized into "blocks". A single block records all of the transactions and confirmations that have taken place since the previous block. Once a new block starts, the current block is "closed out" and added to the block chain.

However, the only way for the current block to end and the new block to start is if someone FINDS a new, valid block. The process of looking for the next new valid block is called "mining". Again, due to super genius math that I won't cover, everyone on the network knows what the next valid block should look like so they can all agree when somebody finds it. But nobody actually knows how to find it except by random chance. Finding the next block takes a ton of processing power, trying random numbers over and over (and over and over... billions of times). If you happen to be the one who finds the next valid block, your reward is to get some coins "out of thin air" as part of that block. When you find the new block and then tell everyone on the network about the new block, they all agree that the first transaction recorded in the block is you adding "new" coins to the wallet of your choice. This is how new coins come into existence in the currency.

The block mining process has two key inputs that are initially set by the original creators of the currency: the block reward and the difficulty factor. They are based on algorithms and consensus within the P2P network, so while the starting values are set arbitrarily by the original creator, they can't just be changed arbitrarily afterwards.

The first key input is the number of coins to reward to the person who finds each block. For instance, when BitCoin first started, the reward for finding a new block was 50 coins. However, everyone also agreed that the reward would be cut in half each time 210,000 blocks had been found. Since there are currently 279,748 blocks, the current reward for finding a BitCoin block is now 25 coins. Once the 420,000th block is found, everyone will agree to cut the reward in half again.

The second key input is the difficulty factor. The goal of mining blocks is to find a new block after a certain amount of time has passed. For BitCoin, the target is for someone to find a new block every 10 minutes. However, finding a new block is based on random chance, and the number of "random guesses" being made is dependent on how many people are mining and how fast their computers are. If the random chance were fixed, then as more people mined with faster and faster computers, blocks would be found more quickly. To avoid this, the difficulty factor is used to adjust what the next valid block "looks like" in a way that makes it easier or harder to find. In BitCoin, the difficulty factor is adjusted every 2160 blocks (based on an initial algorithm and by consensus on the P2P network) so that, based on how many people are mining and how fast they are mining collectively, it will always be about 10 minutes (on average) until someone randomly finds the next new block.

I'm a cryptominer and hash a-way, I SHA all night and I scrypt all day!

The process of finding a block involves "hashing". Everyone on the network knows what the hash of the next block needs to look like, but because of the nature of hashing, there's no way to know how to FIND a hash that looks like that except to just hash random values over and over until you find it. This is the part of the process that takes "work" (computing power) to do.

In a cryptocurrency, the type of hashing being done has a big impact on how "hard" the work of finding the next block really is. BitCoin uses a hashing algorithm called "SHA-256", which has an important trait in that it is not hard "equally". Specialized hardware using ASIC's can be MUCH more efficient at hashing SHA-256 than a typical CPU or video card. An ASIC is basically a custom microchip that some very smart people with EE and CompE degrees have created from scratch. It costs them hundreds of thousands of dollars in research and development efforts to do this, and ASIC-based miners often sell for upwards of $10,000. But they are so much better and faster at mining SHA-256 than a CPU or video card that if you don't have an ASIC miner you might as well not even bother mining a SHA-256 based currency (like BitCoin).

Why does this matter? Well, if you recall earlier I pointed out one key weakness in the cryptocurrency architecture is that if any single entity gains control over more than 50% of the mining efforts, they can subvert the whole block chain process. The more expensive it is to get into mining, and the more custom hardware that is required, the more likely it is that the mining efforts will become consolidated into the hands of a few rich people who can afford the investment. This isn't a fatal flaw, but it is a danger.

In order to avoid this problem, another group decided to come up with a cryptocurrency based on a hashing algorithm that would stay "hard" even on ASIC's. This meant that ASIC's would not have a giant advantage over miners using modern video cards. Since there is a huge pool of people with advanced video cards, and since this pool is not going anywhere since it's not based on cryptocurrency mining but instead on playing video games, then as long as mining on ASIC's isn't much more efficient than mining on video cards, there should always be a large pool of people willing to mine without the high barrier to entry of having to buy specialized ASIC hardware. This algorithm is called "scrypt", and the first currency to use it was LiteCoin.

The benefit is that this keeps the cost of entry lower, hopefully leading to more diversity among miners and reducing the chance that any single entity could ever get more than 50% control of the mining effort. The key difference with scrypt is that it requires a lot of memory, and video cards just so happen to be very fast processors attached to very fast memory. ASIC's designed to mine scrypt currencies still require fast memory too (and a fast memory controller), so it can't be much cheaper than an equivalent video card. Anything that makes memory faster or cheaper will benefit video cards just as much as ASICs. While an ASIC scrypt miner can certainly be a bit more efficient (typically in terms of power usage), it should never become such an overwhelming advantage that mining scrypt currencies on video cards becomes pointless like it has for SHA-256 currencies.

AltCoins

There are now dozens (if not hundreds) of alternate cryptocurrencies in addition to BitCoin and LiteCoin. These are collectively referred to as "AltCoins", and most of them are based on the scrypt hashing algorithm.

The reason for this proliferation of currencies is two fold. One, sometimes there are actual practical reasons for a new cryptocurrency to exist. For instance, someone might have a use case where they want to tweak the block reward or target "time until new block" for various practical reasons. Or the currency might exist to support some other process; for instance, the process of finding and exchanging cryptocurrency can be an effective way to enforce network limitations or access control. There are cryptocurrencies that exist solely so that regular users of a network won't notice the processing power required to mine coins to "spend" on participating in the network, but spammers would have to devote a ton of resources just to mining currency in order to "pay" the cost required to spam the network. So there are some altcoins that exist for practical niche reasons.

The second and more common cause of proliferation is greedy speculation. A side effect of the mining process and the cryptocurrency architecture is that, at the start of a cryptocurrency's lifetime, it is MUCH easier for a single person to find new blocks since fewer people are mining the currency. This means that if you start mining a cryptocurrency very early on, you will get a large number of coins from mining while the difficulty is very low. If the currency then becomes popular later on, this speculative mining can pay off in a big way.

Of course, this only works if you get in on mining a currency very early in its lifetime and if the currency then becomes popular enough to have real value. What better way to be the first to mine a currency than to make a brand new cryptocurrency of your own? And thus, a ton of altcoins are born. Realistically most of these are speculative currencies created by people trying to get rich quick, while simultaneously trying to convince you that there is a real reason for the currency to exist (even though there probably isn't) so that it will become more popular... so that they (the early miners) can get rich quick. As such, you will see a lot of fervor and passion in early cryptocurrency adopters trying to promote whatever altcoin they've dedicated speculative mining efforts into (see, for example, coinye, DOGEcoin, Memecoin, etc.)

Cashing out (or in)

So... how do you turn a fiat currency (like, say, Canadian dollars) into a cryptocurrency? Or vice versa?

More and more, you can often avoid this question entirely. Every day more retailers and businesses are supporting transactions directly in the common cryptocurrencies of BitCoin and LiteCoin. For instance, EasyDNS, my preferred DNS registrar, let's you pay your domain registration and renewal fees directly in BitCoin.

That said, at the end of the day you still need a way to convert cryptocurrency to fiat currency (and back again). To do this you need a bank or currency exchange that supports both. Most exchanges of this kind only deal in BitCoin or, sometimes, LiteCoin. Once you have BitCoin or LiteCoin you can then transfer it to other wallets or other exchanges where you can convert it into other altcoins. Likewise, you can turn your other cryptocurrencies into BitCoin or LiteCoin and then transfer it to an exchange lets you turn that into fiat currency.

Since this exchange will be dealing with fiat currencies, any legitimate exchange of this kind will be doing its best to comply with local laws and regulations for banks or money exchanges. The legal implications of this are kind of up in the air right now in most countries and it's all a bit gray, but if the exchange you use isn't thinking about this aspect at all, chances are they (and you) are in for a big surprise some day.

Because the exchange must deal with government regulations, the exchange that works for you is going to depend in large part upon which country you live in and which country's fiat currency you want to use. In Canada, two major exchanges are CA VirtEx and the Vault of Satoshi. CA VirtEx has a cool feature where you can even get a debit card that works in all standard Canadian debit terminals that draws on your BitCoin balance behind the scenes. Pretty cool! Another popular exchange that supports US dollars is MtGox.

It's important to remember that the actual value of any cryptocurrency in terms of "real" fiat currency is completely dependent on the exchange rate between the two, just like the exchange rate for any two fiat currencies affects their relative value. Because cryptocurrencies are very young and niche, these rates can fluctuate wildly over short time periods. Cryptocurrencies are considered to be extremely volatile compared to most fiat currencies.

Mining and you; a practical guide

So, after all that... are you interested in mining cryptocurrency? Then read on!

1.21 Giga Hashes!?!?

The key metric you need to know for mining is the number of hashes per second you can perform. This is basically the number of "random guesses" per second you are making at trying to find the next valid block. Modern ASIC-based SHA-256 mining hardware is just now reaching several gigahashes per second (Ghash/s) while mining SHA-256 currencies (that's several billion guesses per second). In contrast, hashing SHA-256 on a video card is usually measured in, at best, megahashes per second (Mhash/s)... that's only several million guesses per second. You can see why ASIC mining is practically required for SHA-256 currencies.

For scrypt mining, a modern video card is typically measured in kilohashes per second (Khash/s). A good single video card setup will run from 100 to 400+ Khash/s, and ASIC's are unlikely to improve this number much for equivalent cost.

Profit

The most important thing about choosing to mine a cryptocurrency is to ensure that you aren't losing money on the prospect! If you are going to mine an scrypt-based currency on hardware and video cards you already own, then all you really have to worry about is the cost of electricity. Unless you are paying obscene amounts on your power bill, you are likely to come out ahead in this case.

However, the profitability calculation becomes much trickier when you are talking about buying new hardware JUST for mining. In this case you have to figure out how long it will take for that initial investment to pay off. The thing that is most often overlooked while trying to figure this out is the reality that, as time passes, the difficulty factor of the currency increases over time as more people are mining the currency more efficiently. Trying to estimate the profit (or loss) of buying new mining hardware depends on accurately estimating how this difficulty increases over time.

The previous point is critical, and can make a huge difference. If you could buy one of the most advanced $10,000 SHA-256 ASIC-miners today and have it mining BitCoin starting tomorrow, you could pay off the initial investment in a week and make a killer profit! However, if it takes 3 months for that same hardware to be delivered to you then, estimating with the current average increase in the difficulty factor for BitCoin mining, you would LOSE money. This problem is exacerbated in SHA-256 mining due to the recent introduction of ASIC-based mining hardware and the huge effect it has on the overall mining curve. If you can get your hands on an ASIC miner right now, you can make a killing, but by the time you actually get your hands on one... so will everyone else. And then you'll just be back to barely breaking even.

As a result of this, I'd strongly recommend starting with an scrypt-based currency (like LiteCoin) using hardware you already own. If you can't make a profit mining when your hardware costs are free (since presumably you already have a computer and video card that goes unused most of the time), then it probably wouldn't make much sense to invest a bunch in mining hardware either.

This calculator is a great tool for trying to determine profitability.

Mining Pools

While there was once a time when you could mine a currency on meagre hardware and find the next valid block all by yourself (thus keeping all the reward coins for you), that time has long passed. Unless you are investing thousands of dollars into a giant mining farm, the chances that you will find the next block yourself are pretty slim (and if you ARE investing thousands of dollars into a giant mining farm, you should probably find a better guide than this one!).

Remember, the chance to find the next block is completely random. You could find it on the very next hash, or never. When the overall mining difficulty was very low, the chance that you wouldn't find any blocks was also pretty low, so people could "solo" mine with reasonable results. But nowadays the chance that you are the lucky winner who finds the next block is extremely small.

To address this problem, miners consolidate their efforts into "mining pools". Everyone contributes to the pool by looking for the next valid block. They get a "share" of the pool based on the portion of work they do compared to everyone else. Once someone in the pool randomly finds the next valid block then everyone in the pool gets a split of the reward based on the number of shares they have contributed.

There are TONS of mining pools out there. I've settled on using TradeMyBit for a variety of reasons. First, it has sub-pools for many currency types so you can choose which currency you want to mine. Second, it has a "multi-pool", which allows you to mine the scrypt-based currency that is "most profitable" right now based on current exchange rates. Third, the mining pool servers (called stratum servers) have, so far, all been pretty reliable compared to other pools I've tried. Fourth, they aren't in any danger of reaching 50% mining share; remember, DON'T simply join the biggest pool you can find, you don't want any single pool to reach 50% mining share! Fifth, their fees are very reasonable compared to other pools I've looked at (the fee is simply the cut they'll take out of the shared profits to keep everything running).

Mining Software

Profitable scrypt mining uses modern video cards. If you are mining on an ATI card (Radeon), you want to use cgminer (note: use version 3.7.2 or lower; the 3.8 and 3.9 branches do not support GPU scrypt mining). If you are mining on an nVidia card, you want to use cudaminer.

I'm not going to cover setting up the miner software; there are a ton of guides out there (time to exercise your Google-fu!). However, here is a guide for cgminer on Windows and one for cudaminer on Windows. The mining pool you join will determine which stratum server you use, and many mining pools also have their own setup guides for the miner software.

AltCoin Exchanges

When mining currencies, especially AltCoins of any kind, you usually want to get your earnings transferred into LiteCoin and/or BitCoin. This is especially true if you are mining in a multi-pool setup like TradeMyBit where the currency you mine changes routinely to chase the most profitable exchange rates.

As such, you'll need exchanges that let you convert between all the various AltCoins and LiteCoin/BitCoin. The one I like to use is cryptsy (if you happen to sign up there, please use my referral link, thanks!). Cryptsy lets you setup wallets on their site for a bunch of AltCoins without actually having to download the wallet software for each and every AltCoin yourself. You can also setup automatic sell rules so that as soon as you receive deposits on those wallets they are automatically exchanged for BitCoin or LiteCoin. This works very well in conjunction with a multi-pool miner like TradeMyBit.

The Final Word

Cryptocurrencies are neat, and in many ways represent a far simpler and easier way to move monetary value around the internet in this day and age of international trade. There is even a cryptocurrency dedicated solely to simplifying the exchange of fiat currencies (ripple). As more and more online retailers and service providers begin to support direct BitCoin and LiteCoin payments, the landscape gets even more interesting.

Realistically a lot of the hype and buzz about cryptocurrency was from the speculation, bubbles in BitCoin value, and meteoric rise in BitCoin wealth for those lucky or smart enough to mine or invest in coins several years ago. For the rest of us, though, there is real, interesting, and practical value in the simple nature of cryptocurrencies far beyond this transient hype.