|
| |
|
|
|
|
TCHS 4O 2000 [4o's nonsense] alvinny [2] - csq - edchong jenming - joseph - law meepok - mingqi - pea pengkian [2] - qwergopot - woof xinghao - zhengyu HCJC 01S60 [understated sixzero] andy - edwin - jack jiaqi - peter - rex serena SAF 21SA khenghui - jiaming - jinrui [2] ritchie - vicknesh - zhenhao Others Lwei [2] - shaowei - website links - Alien Loves Predator BloggerSG Cute Overload! Cyanide and Happiness Daily Bunny Hamleto Hattrick Magic: The Gathering The Onion The Order of the Stick Perry Bible Fellowship PvP Online Soccernet Sluggy Freelance SMBC Comics The Students' Sketchpad Talk Rock Talking Cock.com Tom the Dancing Bug Wikipedia Wulffmorgenthaler       |
|
bert's blog v1.15b Powered by glolg Programmed with Perl 5.6.1 on Apache/1.3.27 (Red Hat Linux) best viewed at 1024 x 768 resolution on Internet Explorer 6.0+ or Mozilla Firefox 1.5+ today's page views: 79 all-time page views: 216218 most viewed entry: 2592 views most commented entry: 14 comments number of entries: 563  page created Mon May 21, 2012 06:16:32 |
|
- tagcloud - academics [60] art [8] changelog [40] current events [26] cute stuff [12] gaming [10] music [8] outings [16] philosophy [10] poetry [4] programming [15] rants [5] reviews [6] sport [34] travel [17] work [3] miscellaneous [75] |
|
- category tags - academics art changelog current events cute stuff gaming miscellaneous music outings philosophy poetry programming rants reviews sport travel work tags in total: 349 |
| ||
|
Salvation of Sanctuary The progress on CAPTCHAs has been deferred for good reason, with Team Ham having a world to save; Yes, Diablo 3 has been released, and I along with my cousin and two of his friends availed ourselves of the S$65.90 preorder offer from qisahn (MSRP S$89.90). It came as a boxed DVD, with a brief manual and several guest pass keys. We didn't get off to a particularly good start as Mr. Ham discovered that the Barbarian did not include a hamster commander ( I had to explain to him that that was a different series), and Mr. Robo was aghast at the Witch Doctor not being an actual certified doctor either. After they got over their disappointment, it was decided to pool our efforts with a single Monk character - Oboham.  Lai, lai, mai shy (Click for full screenshot) It all began with a Vampiric Short Sword of Winter (8.4 dmg/s, 4-8 dmg, 1.40 attacks/s, +2-4 cold dmg, 1.20% of damage dealt is converted to life), and Oboham slowly made his way through the four Acts, cutting through the Butcher, Belial and Azmodan in turn (perishing a few times along the way), before facing the Big Bad Diablo himself.  It was kind of anticlimatic The journey took about fifteen hours solo over several nights, which doesn't hold a candle to the Koreans (6 hours as a team), but does include some sidequests. Oboham was Level 31 at the conclusion, which should be about par. A few comments on the game - it sure went by much faster than I recall Diablo 2 being - I don't remember if I ever completed that, having tried it only long after its release. A Monk-specific tip that some may find useful: it is possible to put both Seven-Sided Strike (which has the monk dashing rapidly between enemies hitting them, the same as Dota's Blademaster's Omnislash) and Mystic Ally both as hotkeys despite them being in the same skill category. As far as I can see, there's not too much skill involved, at least for Normal difficulty, with most foes falling to basic attacks combined with Lashing Tail Kick and Seven-Sided Strike to get out of dicey situations, and Breath of Heaven for additional healing wherever required. To make it even more idiot proof, the Transcendence passive turns most techniques into a secondary heal, with Mantra of Healing providing permanent regeneration above all that. All this is extremely welcome with potion use severely curtailed by design. On the plus side, Diablo 3 certainly fulfils the old hunter-gatherer drive of bashing things with clubs and taking their loot, the map levels and cutscene video quality is impressive (if only to be expected from Blizzard), and some of the NPC quips deserve a smile. However, things do get repetitive soon enough (Tristram again, then a Desert City? Okay), and the plot's kind of bland, but who cares about that? I'm not too sure about some of the changes, such as player attributes being automatically assigned (with attribute requirements for items also removed), and characters being able to swop between all available skills as desired, instead of having them chosen permanently in Diablo 2, such that if a different build is wanted, the player has to create and train a new character from scratch, or slowly respec the old character. While attributes can be modified by items, this seems to be in line with a general trend of dumbing games down to cater to the lowest common denominator (i.e. mass market, or auction house as it stands) Thoughts on Dialogue I lost most of Saturday on various errands (where I found out that Hougang may not be a slum, but certainly has a paucity of cabs; I had to resort to placing a phone booking, after which the driver explained that cabbies tend to come out more in the evenings) before rushing down to NUS for the Policy Studies Seminar and Debate Series dialogue session organized by the NUS Students' Political Association, which billed the Acting Minister for Community Development, Youth and Sports, Mr. Chan Chun Sing and MP Dr. Amy Khor as guests of honour. The opening part consisted of a presentation of ideas and concerns from the associated workshop, discussions and debates. I shall first list the complaints and suggestions from the student participants by category, as I recorded them: Transport
Population Dynamics
Housing
Health
Immigration
Without commenting on the above yet, let's move on the the questions from the floor:
With all this said and done, it was time for our Minister to give his thoughts - but first, he asked how many in the audience had participated in the earlier workshops etc, by, wait for it, raising hands... I suppose more politicians *could* do with signature entrance themes, like wrestlers That formality done with, Mr. Chan then began with the observation that policy making is complicated, with often orthogonal and conflicting ends, unlike the private sector where the one KPI is profits. Therefore, governments have to be clear about policy objectives, and beware of the Law of Unintended Consequences (somehow, he hesitated to use the aphorism "the road to hell is paved with good intentions", believing it possibly offensive) As an Economics major (too), he noted that it is wonderful to study economics, and that the basic problem is satisfying unlimited wants by finite means; he then asked for desired ends for Singapore (noting that the size of the population is a means and not an end), and got such responses as quality of life, a cohesive society, job opportunities and freedom of choice and expression, and noted that to achieve X objectives, one generally requires X instruments, easing into the Socratic method for the remainder of the dialogue. It shouldn't take too much to guess that he basically toed the party line, along with having a great time shooting down the weaker (cute-cute) ideas (e.g. if one can afford a significant downpayment, one would probably not need a scholarship), making assertions (if foreign scholars are to be dealt with harshly for breaking bonds, so must local ones!) [N.B. Actually, why need to like that ah?] and expounding upon accepted wisdom - that Singapore as a city-state has to remain a Tier I megacity and compete against others like Shanghai, Seoul and Tokyo, and to do that it requires both quality and quantity (i.e. more people) In particular, he says, as a small country Singapore has to be nimble and fast - companies will gravitate to China whatever its growth rate, but Singapore has to maintain a higher growth rate to stay competitive as a price-taker internationally. The last time we experienced slow grouth (2001-2005), a third of the people had negative real income growth, which was arrested only by the recent infusion of foreigners. [N.B. While the correlation may exist, the question remains of how long this artificial boost can last for - ten years? twenty? And then back to square one? And if TFR among the newcomers is in the doldrums too, then what?] He does make the good point that a higher Gini coefficient may be unavoidable for Singapore, since as a city-state there is no countryside for less-productive elements to retire to. On housing, he states that Singapore does meet the international benchmark of paying 20% to 25% of income towards one's residence [N.B. Though it should be remembered that it is towards a lease and not actual title], while refuting claims that the percentage of income needed has increased over the years, using his mother's experience as an example (S$62k on a S$100 a month salary). Thanks to Mr. Chan, I can also now confirm that the HDB is running a half-billion dollar deficit annually [N.B. Though likely not due to actual construction costs, but whatever land is said to cost] Indeed, Mr. Chan argues that we are actually more socialist than the socialists, pointing to higher relative subsidies for one and two-room flats, and a S$20k/year average subsidy for students, and cautioned over overreaction to housing prices, nothing that the financial crisis in 1997 created a huge surplus. It might be more interesting to note what was not covered in detail - raising the fertility rate ("we'll try") [N.B. As noted many times before, I'll be surprised if they succeed; However, having said that, it's not all the government's fault - most of the actual causes are probably politically incorrect], locals who perform as well as (most of the) foreign scholars not getting the same reward ("we'll fix it") transparency and quality of immigrants, to give a few. Dr. Khor gave an update on the hospital cluster system towards the end, before various prizes were given out for the students. I had originally planned to drop by the "That We May Dream Again" event after that to provide a counterpoint, but it has been postponed to 2 June due to the by-election; I shall therefore reserve a fuller treatment of the events until then. As it happens, Mr. Chan's parting shot was to "Never stop asking!" - maybe, he will not be disappointed.
- academics - It's the final day of the Premier League, and the current punting standings are: 3207 seeds for Mr. Ham! (both from 3100 seeds wagered) Mr. Ham: Useless human is going to lose after I gave him a thousand-seed headstart! Nyeah nyeah! Me: I'm not certain that FAKEBERT is human... Mr. Ham: I'm not letting that stand in the way of my declaring hamster superiority. Final wager: What better way than to finish with a flourish by having hams share the spoils? Tottenham to draw Fulham! (at 4.25) FAKEBERT: I'll take Manchester City to beat Queen's Park Rangers (at 1.10) Me: With that bit of business out of the way, may I draw the focus back to one of my part-time diversions, the study of machine recognition in general and CAPTCHA solving in particular, which I have given attention to not too long ago. However, while that more applicative-based attempt was centered on a relatively simple target, it is time to widen our horizons; and what better way than to set the Google CAPTCHA in our sights? Less than half a year ago, Stanford researchers reported that they were unable to break a single of Google's CAPTCHAs, yet were able to crack 13 out of 15 other types of CAPTCHAs with accuracies that are usable in practice (mostly over 25%) [see academic paper] It should be noted that a researcher with links to a local lab had reported [see paper] the ability to solve 68% of Google's CAPTCHAs back in mid-2011. However, this success may be tempered by the revelation in Section 3 that they initially divided the CAPTCHAs into "user-unfriendly" and "usable" classes, and developed (and presumably evaluated) their method on only the easier "usable" class, using 100 samples. Clearly, the conclusions may not then be applicable to the wider set of all CAPTCHAs, which is what users actually face. To this end, I requested Mr. Robo to collect the data for an independent experiment. Mr. Robo: Aye, aye. I have obtained the sum of 2000 samples from the former Google Accounts UnlockCaptcha page back in February. They've since closed this source of CAPTCHAs, but it should be valid for our purposes. Me: Nice work, Mr. Robo. I'll see you get that promotion to code bunny soon. Well, the obvious next step is to create a ground truth for these CAPTCHAs. And how might we go about that? *Looks at Mr. Robo, together with Mr. Ham* Mr. Robo: Wai... wait a minute! It's written in my contract that I don't have to do data entry! Esquire Pants: *appears out of nowhere and inspects contract* Looks legit to me. *Looks at Mr. Ham* *Mr. Ham stares back blankly, slowly keels over* Mr. Robo: My word! Is he dead?! Me: *pokes the motionless Mr. Ham with a chopstick* He's certainly getting better at it, he's even getting the turning black part right. Oh well, I suppose it can't be helped... *Hours later* Me: That's... that's all two thousand CAPTCHAs sol... solved. I... I don't want to... look... at another CAPTCHA for the rest of... my life. Mr. Robo: Erm, but how would you know how confident you should be about your answers? Me: Tha.. that's right... so... how do... we... resolve th... this? Mr. Robo: Um, there's intra-rater reliability. Which basically means you got to do it all over again, and see how different the results are.  My reaction to this realisation (Source: gifbin.com) *More hours later* Me: *crawling up from under the desk* D... done. Mr. Robo: But wait, what are you gonna do if they still don't agree? You had better do it a third time so that you can apply a majority voting rule for the disputed cases. Me: I... will... personally... strangle... anybody... who... suggests... that... Mr. Robo: Um. Okay. Never mind then. Let me analyse the results first. *codes away* Lookie what we have here, only 143 of the 2000 answers disagree with each other, giving a 92.85% intra-rater reliability for you. A 2010 study, again by Stanford, found 92.1% agreement by two out of three humans offering their services under Amazon's Mechanical Turk framework, and only 66.72% unanimous agreement by all three, for Google's CAPTCHAs, to give some perspective. Bottomline is, solving state-of-the-art CAPTCHAs isn't that easy, and may even have some correlation with general intelligence - the paper noted that solving speed is weakly correlated with educational level, 9.6 seconds on average for those with no formal education and 7.64 seconds for those with doctorates, at least for image CAPTCHAs. Some have even declared CAPTCHAs dead (back in 2008), and even the better ones are often regarded as near-unreadable (see comments here, and feedback like this, which has a good point about international users and Latin letters), with an obvious solution being to re-engineer sites such that the onus of stopping bots falls on the server side (e.g. using spam filters), rather than the client. It should also be noted that the results obtained from our dataset might not be directly comparable with prior research (again!), since Google likely updates their generation algorithm every now and then. For example, from our 2000-sample dataset, the average length of a CAPTCHA is 8.98 characters, with a median of 8, a minimum of 5 and a maximum of 11; this is a departure from the paper covered previously, which states that the string lengths vary between 5 and 8 characters only, at least for the "usable" class.  In fact, Google CAPTCHAs of length 7 or less were extremely rare in our dataset, with the vast majority being of length 8 to 10. Google might well have decided to bolster security by increasing length, since it is evident that each additional character should reduce machine solvability by at least (1-x), where x is the average recognition rate for each individual character, assuming they demand perfect answers. As it turns out, CAPTCHAs for which there was intra-rater disagreement tended to be slightly longer, but not significantly. The trouble, as I will show later, lies elsewhere. Me: The problem remains. How do we proceed? Mr. Robo: Well, it would be best to collect some more data first, so we don't depend on answers from just you. Do you have any favours you can call in? Me: I think Mr. Ham is the go-to guy for that. *looks over* Nope, he's still dead. Mr. Robo: Oh, he was up and about munching on some peanuts just now. Me: Forget it, I'll put in a plea to my friends, especially those who are currently pursuing or may in the future pursue further studies, since they are more likely to be sympathetic. I can't quite ask them to go through all 2000, so please select a hundred examples for me, split evenly between my intra-rater agreement and disagreement classes. *A couple of days pass* Me: Alright, it seems like I got three responses. So how did they do? Mr. Robo: More detailed results are available, but I will give a summary. Of the 50 disagreement images out of the 100, 13 (26%) of them were agreed upon by all three volunteers - for these, I suspect that the disagreement was more due to fatigue and other factors on your part, and not because of any inherent indecipherability of the CAPTCHA itself. As a whole, the agreement class did still obtain much better consensus among everyone, as is to be expected. Fully 66% of the images in that class were agreed upon by all four participants, as opposed to 26% for the disagreement class, picking either of your answers as valid for that class. The distribution is as follows:  It therefore does seem that your intra-rater disagreement is a pretty good predictor of how difficult the CAPTCHA actually is, and the 60-odd percent concurrence tentatively supports the Stanford findings. Four of the disagreement CAPTCHAs (8%) received a different solution each of the five times they were evaluated, and here they are:  urwledsh - vawledsh - urveledsh - uawledsh - vorveldsh pnoatemon - proatemon - prxtemon - pnatemon - pratrnon immelxino - immelixino - immerxino - immeixino - immerxno unmvanevo - unmvnero - uraminero - uamwnew - uvaraunevo It therefore seems near-certain that some CAPTCHAs are, indeed, basically unsolvable, or more technically, that their deformation has destroyed critical information. From observation, these "bad" CAPTCHAs are especially wavy and squashed, making it difficult to tell loopy (e.g. m,w) or thin-type characters (e.g. i,l) apart. Some tangential statistics: Of the 50 agreement CAPTCHAs, you have an average of 84.67% agreement with the volunteers (92%, 86% and 76% respectively). Volunteers 1 and 2 agreed 84% of the time, but their agreement with Volunteer 3 was only about 70% each. On to the speed of solving - the volunteers were told to solve the CAPTCHAs as-per-normal, as if they were confronted by them in the normal course of web surfing. Since they could well have taken breaks in between (which you can doubtless understand), I report the median time taken. Note that since individual times were recorded to the second, and internet speeds may have had an effect, this is a very rough estimate:  It is observed that Volunteer 2 [Median time: 4s/Mean time disregarding outliers: 4.94s] was the fastest, followed by Volunteer 3 [7s/8.28s], while Volunteer 1 took slightly longer [8s/8.36s] (but also achieved the highest agreement with yours truly); the average of about 7.2s is slightly faster than the 7.64s reported for solvers with a doctorate, even considering the skewed nature of the 100-image dataset, which contains harder CAPTCHAs than might be expected. Somewhat surprising is that the collated results do not show more time being spent on the trickier disagreement CAPTCHAs than the more straightforward agreement ones. I'm sure you can cook up several hypotheses for this:  Me: Enough with this, anything actually useful? Mr. Robo: About that, one realisation is that the strings are far from truly random - some substrings, in particular, cropped up every so often: "ssess" about 22 times, for example, although that might be expected only in no more than one in a million CAPTCHAs were the characters really selected randomly; Even "glys" came up about five times! We can very quickly confirm this through, you guessed it, frequency analysis, which despite some errors should give the big picture (17967 characters from the 2000-CAPTCHA data, Concise Oxford data from Wikipedia):  It seems safe to say that the CAPTCHAs were generated from a distribution similar to an English corpus. Indeed, this may well aid CAPTCHA recognition for humans - instead of having to identify each character on its own merits, neighbouring characters might well be used for difficult characters or substrings, if subwords are incorporated, a feature which many other CAPTCHA implementations may have missed. Indeed, an update on the baboon word recognition study mentions that they may well be making their decisions based on bigrams (letter-pairs) or trigrams (or not)! Does the CAPTCHA data in fact conform to common bigram distributions as well?
As it happens, not quite. For example, "th", which is the most common English bigram by some distance, is not represented within the top 30 CAPTCHA bigrams (it comes 86th); however, there remains quite a bit of overlap, with 23 of the 39 most common English bigrams found within the top 30 CAPTCHA bigrams, which again suggests the use of an English-like corpus. Alright, I think that's enough for today. Me: Hey, you haven't even began to crack the CAPTCHAs proper! Mr. Robo: Patience, Hamopolis wasn't built in a day.
- current events - - so says the leader of Singapore, who evidently also believes that the election should be on local concerns, like how not to be ignored for taxpayer-funded upgrading I had to suppress a slight smile at this statement, which comes across as especially poignant with the uncle of his Hougang by-election candidate having once resigned as an MP right before pleading guilty to a cheating charge (which happens to actually be against the law, unlike knocking up a colleague), in response to which the PM praised him for doing the right thing - perhaps he expects the same plaudits for misbehaving members from his opposition counterpart? It so happens that this MP later got hauled up for taking some S$10k in bribes (which seems a little disappointing for a man of his pedigree), to which his former party has been conspicuously silent to the best of my knowledge. Not that the sins of the uncle should be visited on the nephew, who as our resident political analyst has sagely forecast back in February, will be the PAP's candidate. As the by-election has been set for May 26, with a 50 year-old veteran of six years being the WP's choice and other parties vowing to stay clear (disregarding those itching to lose their deposits), four of Herr Ahm's five predictions have come to pass; it remains to be seen in a fortnight whether Herr Ahm does the grand slam with a vote return of 63±3% for the WP. Tying up the loose ends, the court case on whether the PM does in fact have unfettered discretion on whether and when to hold a by-election has therefore been withdrawn, having done its job, which has probably allowed some of our best legal minds to sleep more easily. They might be called into action soon, seeing as to how the academic referenced in the previous post has stepped up his probing of supposed fundamental irregularities (known to be unproductive for locals with loudspeakers), even going as far as to answer queries on a popular local forum. Notch simpur, as regulars of that forum might say. Personally, I can easily buy the "enormous unreported holdings" a.k.a secret strategic reserves hypothesis, the existence of which is hardly classified. As to how much is actually in it... well, if we knew, it wouldn't be secret any more, would it? I won't be counting on more details from the horse's mouth anytime soon, and can only comment that the recurring furore over this issue is a matter of trust, in particular that citizens should trust without direct evidence that quasi-autonomous bodies will be scrupulously honest about the behaviour of extremely large, non-publicly-documented sums of money. Though they might as well, seeing as that they may not get the figures even if they get elected as President. Fact Follows Fiction The sabre-rattling between China and the Philippines over prime real estate in the South China Sea has gotten loud enough that the former has found it necessary to deny that it may start a war over the territory (while demanding protection for its nationals). While the outcome of a one-on-one armed duel is likely not in doubt, the situation is complicated by the Mutual Defense Treaty that the latter has with the United States, which the US has confirmed it will honour (to an appropriate extent) Now, Obama might view a khaki election as being to his favour (and he would probably be right), having milked his bin Laden success heavily, but the guys he'll be up against this time aren't bearded fanatics holed up in mountains, but a nuclear-armed adversary with a huge chip on its shoulder, a two million man army and their Walmart supplies, which should give his administration pause. As for the backstory, it's nothing new - the dispute over the area has simmered for decades, with Brunei, China, Malaysia, Taiwan, Vietnam and the Philippines having overlapping claims; as such, even desolate scraps of land such as the Spratly Islands (Nansha Qundao to the Chinese) have become bitterly contested. All this bother over blue water might appear a bit overwrought even by the standards of political ego, but for the possibility of oil and gas deposits. That explains a lot.  Competing claims illustrated (Source: wikipedia.org) From the graphic, China's claim extends a long way from its mainland, and would appear rather unsubstantiated had it not been bolstered by arguments of historical sovereignty over Nansha. Going further back, the oceans have seldom been a source of contention for their own sake, trade routes and privateering notwithstanding - mariners had enough on their hands navigating the vast featureless deeps, and rendering assistance to fellows in distress, regardless of nationality, is a time-honoured unwritten code among men of the sea (which might however be reconsidered if those fellows are waving rocket launchers about). Unfortunately, improved monitoring technology, the acknowledgement of fish as a limited resource and of course, offshore oil have turned all that on its head. Pure technocrats might think it elegant to simply allocate every patch of water to the country whose land it lies closest to (a Voronoi map [again] of sorts). It transpires that the regions above do roughly correspond to such an allocation - weighted by the military strength of each nation. As it pans out time and time again, if you can't keep it, it ain't yours. On to the fiction, something close to this very scenario has been covered in Dragon Strike (set in 2001), which I happened to pick up back in JC. Penned by two journalists, the book describes how a military resolution might unfold, although the opponents envisioned are the Vietnamese, not the Filipinos:
If only it could all be settled with evidence from a blog! Happily, Herr Ahm sees no big moves, but a lot of rhetoric, for the near future, after which China basically swallows it up. Full On Conspiracies! (More Spoilers!) I chanced upon Dan Brown's The Lost Symbol in the library, and learnt that symbologist Robert Langdon, after tangling with an Illuminati master (who wasn't really one, being actually the son of the Pope, whom he murdered by poison) through one novel, and an Opus Dei albino numerary (who also murdered his father for killing his mother) through another, comes up against a Freemason renegade. Somehow, I was not overly astonished when that renegade turned out to be the son of the 33rd Degree Grand Master that he had kidnapped to kick events off, although there is an inversion in that instead of trying to kill his dad, he instead wanted his dad to sacrifice him (shades of Isaac here)  Oh yes, Langdon gets one improbably attractive muse per book too (Source: thecomicninja.wordpress.com) Utterly predictable puzzle-geocaching plot structure aside, the writing style is an acquired taste often roundly shredded by critics, though it is worth considering all of them put together have probably not moved a fraction of Brown's 80 million copies - for all their complaints, being readable is an art in itself, and Brown, like Archer, has apparently mastered that. [N.B. I always wonder what our literary canon would be like if approached without preconceived notions - put a hundred otherwise well-adjusted students with no background in a well-stocked room, and I wonder if they would be able to pick out the classics. Same for art and music.] Picking controversial topics doesn't hurt either, and neither does taking some liberties [N.B. some supposed "errors" may not actually be errors] with facts, and one of Brown's best skills is blending them such that his inventions become plausible (was Newton's 33-degree scale influenced by Freemasonry?); some unlikely ones turned out true, such as the original meaning of atonement (to be "at one"), and George Washington depicted as ascending to godhood inside the dome of the US Capitol Building (his statue of the same has been relocated) - it may come as a surprise to certain sections of the population how many of the Founding Fathers were Deists or Freemasons (see case for Washington; then again, few begrudged future Georges their own secret society)  Presidents and their images may not concur (Source: cracked.com) Brown does make the good point that Freemasonry (mentioned in passing before) is hardly that shadowy, and indeed application for membership is generally open to all (males - in 1959, some 4% of the eligible US population were Freemasons!). Indeed, Singapore has lodges too, perhaps more of them than might be expected (in a wholly unexpected connection with the previous section, the Horsburgh Lighthouse Foundation Stone on the once-disputed Pedra Branca makes reference to a "Lodge Zetland"). He does also posit that there are "circles within circles", though. Another fact I thought he was certainly making up was that the Pigpen cipher is Masonic in origin, which it turned out to be. I suppose people weren't big on cryptanalysis back then (Poe did wow America with simple frequency analysis). A maybe-entertaining task for the bored: given a collection of valid (say, credit card) numbers generated by some scheme, design a method to postulate new numbers of the form (satisying any built-in checksums). [N.B. Note that this may be what keygen coders do.]
Specs Savers It's been a long time, but it's finally done:  Copper wire and bondo, outperforming cable ties since 1950 Oh, and the blind are being healed... by science. The Search For Novelty It may sound bad, but I honestly feel dumber as the graduate course winds on; apparently the realisation of what one doesn't know grows much faster than what one gets to know. I've got to think about how to get out before I regress into a gibbering idiot, which should at least raise my social stock. The trouble is coming up with something sufficiently novel, as the more I read, the less I believe that anything new (at any rate, new and approachable) has ever existed. Oh, there's of course putting a twist on prior art, or squashing them together, such that they yield some slight improvement on some special case that will be argued to be more general than it really is - which is probably what I'll end up doing. More bashing pieces into new configurations, then (though it may be one of the last to be discovered without brute force...) On research, fashions do come and go, as one Google employee noted back in 2005, even before when I toyed about with Google Trends (yes, it's still around). I figured I could do a quick update using Google Ngram Viewer, which puts the statistics of all those books they aggressively scanned (sometimes without asking, but hey, they are data incarnate) at everybody's fingertips. Note that the occurrences are normalised by the quantity of books published each year:  As is confirmed, while references to "machine learning" are steadily (if slowly) growing, "expert systems" fell off a cliff towards the end of the 1980s, probably about when people began to realise that they were basically huge and unwieldy decision trees. Neural networks too hit a hump towards the end of the 1990s, to the point that one of its founders observed wryly that new researchers who came in after 2000 may not have heard of it (it does probably draws its base from further afield than just computer science), while genetic algorithms have kind of plateaued. [Notably, there exists a strange hump for these disciplines between 1900 and 1910; either someone was seriously ahead of his time, or some publishers felt it safe to omit the first two digits of the year. Then again, Mr. Robo assures me that all these are basically interchangeable, as they should all be transformable to and from, say, a (possibly complicated) decision tree. I'll take his word for it.]  The increase in mentions of support vector machines have not shown any sign of slowing... yet (though they are still a blip in absolute terms), the same for random forests, to a smaller extent. There is, certainly, debate as to which of these is best for a certain domain (of especial interest to those who actually need concrete results), but this will probably never be fully answered, in the sense that whether some selected subset is representative of a domain is probably already impossible to prove; it is true, though, that a simple method used properly may well be better than a sophisticated one used poorly (SVM and parameters, I'm looking at you) [To be continued with an applicative example... soon.] The House Debates
Me: Well, well, they haven't been too bad of late, coming to an agreement to allow private jetties at Seletar, pending their refurbishment; it's all these little things that stick in the mind. And the union fellows have finally remembered their job scope, even if it's just talk. And a think-tank worth the name has realised the basic logic that pumping people in (while shouting down dissenters with accusations of xenophobia) without fixing the birth rate is equivalent to kicking a grenade down the road. Things may finally be looking up! Mr. Ham: Seeing as to what being here does to the urge to procreate, importing cadavers might have a more desirable demographic outcome in the long term. Me: Don't be such a hater, they even made a profit from selling stock in Chinese banks! Granted, part of it can be attributed to being a sweetheart pre-IPO investor, but I'm glad we got the better of the market for once. Mr. Ham: You might want to reconsider the "we" part, though in retrospect even they couldn't go wrong with CCB, which as we have seen maintains undiminished demand whatever its condition and price. For a small fee, I will have Esquire Pants write in to the papers to claim proper credit and recompense for the investment strategy employed. Me: No need lah, no need lah. Mr. Ham: Continuing on, there may be something in the water at the Ministry of National Development - why else would they claim that HDB flat sizes have not been shrinking, when they candidly admitted to it six months ago? They might consider hiring a consultant dedicated to keeping the spin straight, and I know just the ham. ...and why am I saying all the bad things today? Me: I don't want to lim kopi mah. Mr. Ham: Me neither, I much prefer tea... oh. How about let's talk about a principality called Hamopolis instead? Me: Hamopolis, sure. Mr. Ham: Now this place Hamopolis, its public transport system is falling apart, very coincidentally like here, though there is otherwise completely no connection between them, of course. There have been calls to nationalize it in response... Me: Like here, again? Mr. Ham: ...but the catch is, in Hamopolis, the citizenry doesn't realise that their system is already nationalized! Me: Huh? Mr. Ham: Let me explain with some simple infographics. This, I believe, is what most people have in mind when they think of "nationalization":  In this model, the relevent government ministry builds and administers the system, which likely requires a huge (but justified) initial investment, which happily for Hamopolis is not a problem due to their mighty reserves (which came from taxes in the first place). In an ideal case, the system is funded and maintained by a mixture of fares (and taxes if necessary) such that it runs without a deficit or surplus in equilibrium. The people in charge do their jobs and draw reasonable wages for the not overly taxing task of administering a monopoly of an extremely inelastic good. Me: But won't there be inefficiencies without incentives, leading to oversights like overpaying on quality parts or maintenance? Mr. Ham: About that, observe the Hamopolis model:  Marvel in the beauty of the setup: Ministry T, which sounds like it should be in charge, has next to no say in practice, but has the job of appealing to Ministry F for financing when things go pear-shaped, which Ministry F has to approve after grumbling since, like big banks, public transport is too big to fail; but that's not a problem, Ministry F controls an external investment company which runs the system in the first place, and that brave paragon of competitive capitalism rakes in huge profits, year on year, which it then invests very wisely. Simplified further by removing some unimportant links in the chain and focusing on the inputs and outputs:  Me: Shouldn't the people of Hamopolis be mildly annoyed? Mr. Ham: Well, they do have the consolation of an extremely successful investment company that they can be proud of. And there's more, they did it without touching any pension funds:  So, as you can see, the investment company definitely does not manage any of the pension money. The money went to the board, which did the responsible thing and bought bonds from one of the most stable governments around. All this money was then locked in one vault, while some other money was handed by the government to the investment company. No connection at all! Me: Um, that isn't too convincing... Mr. Ham: I thought you had a degree in economics? This is basic stuff I learnt in my first year at Thug U - after you get funds from some source, like loansharking, all you have to do is to pump it through some legitimate business, like the convenient neighbourhood integrated resort or laundromart, and there is then absolutely no link between what sits in your bank account and the original act! Me: Still sounds risky to me. Thankfully, that could never happen here, since a minister helpfully dumbed down the relationship to us and clarified that no CPF funds go to our government-linked investment companies. Mr. Ham: You know, a China-based academic recently published a research note accusing one of those companies of achieving unrealistic Buffettesque annual returns of 17% over 30-odd years, in view of the fact that most of their investments have been domestic until recently, and the domestic market return has been far from that. Evidently he hasn't taken into account the value of all the former state assets that just happen to fall into their lap every now and then as and when required; all that included, I would be disappointed if they didn't hit 17%! Flip Flops Me: Whatever lah, I think we can still trust the government. *glances around* Mr. Ham: Not like you have much of a choice there. From my perspective, them trying... things may not automatically be bad - but they had better be an Ashley Young going about it, instead of being Andy Carroll:  ...can't even give one point for that (Source: whoateallthepies.tv) Me: I hope, I hope. For FAKEBERT (3198.5/3000 seeds), Newcastle (+1.5) to stick it out against City (at 1.50) Mr. Ham (3027/3000 seeds): Making it easy, Fulham to beat Sunderland (at 1.80)
- academics - It seems right to do stuff on Labour Day, so here goes. Like That Lor One of my profs posted an article on the shortage of good computer science and engineering graduates, which noted among other things that although CS has very low unemployment rates, the U.S. in fact graduated more CS majors in 1984 than in 2009 (and even when lumped together with mathematics, it doesn't crack the top 100 most popular majors there - one university has even eliminated its entire CS department!) [Errata 15 May 2012: The not-in-top 100 major bit probably isn't true. Mr. Ham apologizes] Fortunately for me, CS doesn't seem to be on the chopping board here yet, with the fellows at the top even kind enough to share (some of) their vision with us, and one polytechnic graduate even went as far as to create an online appeal to get into the programme; I can't help but applaud that go-getting spirit (of course, it may get old if everybody does the same, and there's also another side to the story) [N.B. On the topic of admissions, the NUS management department top graduate for 2011 (sic) has asked the Prime Minister on his Facebook page the reason why he was not only not admitted to a government ministry job, but had his ideas plagarized by the ministries (discretion doesn't seem his strong suit)! I wonder if the PM had expected this sort of feedback when he took valuable time off not holding the by-election for social network outreach, but quite apart from government hiring practices, why the heck is a top business graduate aiming for the civil service in the first place? Well, at least it didn't get to this (though it's almost certainly a joke)] Of course, the reality isn't all roses; it is common knowledge that there aren't that many really high-paying jobs in the CS field here, with services like finance and real estate still all the rage - why slave away over a terminal, competing against imported budget coders (see, even al-Qaeda is doing outsourcing now), when one can just smile a lot and earn tens of thousands per completed transaction (or at least that's the perception)? [N.B. Here, I have to rant about too many people wanting to get rich far more than wanting to actually do anything well. While I'm not completely against the acquisition of wealth, I have never managed to rid myself of the feeling of slight unease I get whenever I hear the phrase "passive income" (at least the do-nothing-special sort; frankly, I was sort of tempted to just apply for a HDB flat with a relative and thereby contribute to the growing bubble, but other than this being perhaps no longer possible, that boat may have sailed) - as Donkin often returns to in The History of Work, work does mean something... right? No, pumping and dumping penny stocks doesn't count as that in my eyes.]  (Original source: thefinancialfreedomfoundation.org) The catch is that to earn a good to great salary as a programmer, one generally has to actually be a good to great (and probably creative) programmer, and sad to say a degree in CS is not an ironclad guarantee of that (one memorable post mentioned that 199 of 200 applicants for programming jobs can't write code at all, which even if exaggerated tenfold, is still scary - this divide simply can't be charmed its way out of, unlike with some other jobs) [N.B. There are horror stories about interviewees, after having presented impeccable credentials and talked up a storm, not even being able to swap two variables; however, swapping variables without using a temporary variable seems a common enough (if probably impractical) interview question. There are several ways to achieve that, the easiest probably to let x=x+y;y=x-y;x=x-y, although there are more mindbending approaches like the XOR swap. Personally, I wouldn't put too much stock in such "tricks", cool as they may be - code may not be maintained by the original author!] Part of the trouble, I gather, is that programming can be, make that is, unforgiving - if it doesn't compile, it doesn't, and if it compiles but produces incorrect behaviour, it is still wrong (or, if the programmer is feeling cheeky enough, a new feature). Contrast this to some other fields, where everybody's opinion must be correct in its own way, inconvenient details like displeasing facts be damned. Data Play The official LIBSVM tutorial describes grid search as naive but straightforward, psychologically safe and trivially parallelizable. The caveat that it itself remains a heuristic that relies on the response of the dataset to parameters being reasonably uniform should still be kept in mind, however. One can imagine a grid search of normalized unit length in each dimension returning the best accuracies in a region around the examined point [Ca,ya], but having the best parameters actually at [Cb,yb], contained within the unit square defined by points [C1..4,y1..4] which all individually return accuracies worse than [Ca,ya] and which therefore may not be discovered by the search. Of course, this intuitively could be resolved by using a finer grid and exhaustively examining more points in the (infinite with real-numbered values) parameter space, but this would take a longer time, and as the time required grows quadratically each time grid cell length is halved, cannot be kept up forever by empirical means. From this observation, the question naturally follows: For any dataset, is there any way to (even restrictively) prove that some initial granularity of search (grid here, but can be extended to any bounding method in general) is sufficient, such that the true optimal parameters are guaranteed to be obtained within, i.e. the search must converge to the optimal parameters?  It was kind of regular for the cells, thankfully Offhand, the answer seems to be no (excepting bounding the entire set), at least without further assumptions on the structure of the dataset. Let there be some dataset D for which grid search at some granularity G fails to uncover the optimal parameters (it might be thought of as the fine texture, or frequency, of the dataset being too small relative to the grid). Then for any given granularity G', a corresponding scaling-and-tiling R' of the original parameter response function R can be found, such that G' again does not converge to the optimal (phew, that was long-winded) Happily, there are quite a few ways out of this. By far the most common is probably just to ignore all this and just pick some G (as I did). Farmers may not be able to prove that their agricultural technique is optimal, but that's no reason to starve, right? Those creeps are there to be last-hit, after all. I gather this does often produce good-enough solutions, and one can always examine the response over the grid to get a sense of the distribution (yes, it doesn't always work) Another approach is perhaps to try and construct the system such that for each set of parameters, it provides a guaranteed upper and lower response bound for all points in an area around it, and therefore guarantees converging search; this is sort of related to minimax trees, with the complication that while trees in practice have a finite (often very small) number of branches, porting this to parameter space would require some proof that holds over (the infinite number of points in) an area. Related to this is Wolpert's celebrated No Free Lunch theorem, which states that any algorithm that performs well on one problem must pay for it with poorer performance on other problems, such that their performance is in the main identical. However, this conclusion appears to follow from the observation that in the space of all possible problems, almost all of them are extremely random, and therefore no algorithm can be expected to perform particularly well on them anyway; since many scientists (and certainly businesses) are more interested in solving interesting problems, and not all of them, Wolpert's warning (which seems to have its roots in philosophy, a grounding in which does pay) would then exist as merely a curiosity to them.  The BABOON algo achieves about 70% for word recognition (Source: wikipedia.org) If we put aside the question of whether induction can actually be justified in practice, then an optimal solution can be defined (whew) - if the true distribution of a feature is known for multiple classes, then the minimum possible error, the Bayes error rate, is the amount of overlap between them - no algorithm can be expected to do better than that over the long run, which implies that if your features overlap greatly (and are independent), the only way to improve classification performance is to... find better features. [N.B. Repeat warning: the true distribution may well not be Gaussian, though it seems popular to model everything as one by taking the mean and variance of training data! Well, if it's good enough... also, some distributions simply don't fit into conventional machine learning - try classifying the prime and non-prime number classes by proximity metrics only, for example] So, is there some way to reach this optimum? In general, no, since we can only estimate the probability distributions on natural data. However, it is comforting to know that at worst twice the Bayes error rate can be achieved with the K nearest neighbour (knn) algorithm (which for 1-NN can be illustrated by Voronoi regions, seen here previously). And then the bad news: this guarantee only applies when the amount of data approaches infinity, and the correct parameter k has to be found. Okay... [N.B. Interestingly, while one might then imagine k-Voronoi diagrams as the corresponding classification map of knn, its common definition isn't quite like that. While we're on points, I might as well mention the extremely beautiful closest pair algorithm, which non-intuitively proves that one does not have to inspect every possible pair of points to find the closest pair; then again, in practice, if the number of points (say, nodes in a network) is so great that a simple exhaustive search would take too long, I suspect that many coders would just fall back on random search, which should give acceptable results with high probability quickly] Well, as knn happens to be Mr. Ham's favourite technique (he says it reminds him of his time at gangster finishing school, where he spent many happy hours learning to chain vulgarities together in soporific magnificence, obeying strict rules of metre and rhyme), I spent a bit of time implementing it (another of its attractions is that it can easily be coded from scratch, even on a desert island - but hmm, where would the power source come from then?) and trying it on the cell data (note that each feature has a value between 0 and 1, and thus has approximately equal influence on the "distance"):  We see here that knn does not quite reach the heights (dotted line) that SVM did, which is kind of expected. Interestingly, the best results are obtained when k=1 (i.e. each cell is classified according to which other single known cell is closest to it in feature space) and k=2, depending on validation approach, which seems to indicate that fitting tightly is the way to go for this dataset (supported by good regularization parameter values found to be high for SVM, which has its tradeoffs) Mr. Ham: KNN! When an additional 22 noise features, each randomly assigned a value chosen with uniform distribution between 0 and 1, are added to the 22 real features, leave-one-out accuracy drops to a poor 0.5270 at k=1, and 0.5548 at k=2. This should help persuade that bad (non-discriminative) features can easily detract from performance in practical applications. And so how good are the chosen real features actually? Inspired by linear discriminant analysis and Otsu's method, a crude estimation of the usefulness of each feature was obtained by computing the average between-class mean differences (which suggests how well they are spread) and average within-class variance (which suggests how tightly values are distributed within each class) of each feature:  Between-class mean distances (larger should be better) The best features by this metric are, in order, the number of small bright blobs (dots), probably due to their effectiveness in separating centromere cells, the distinctiveness of large bright blobs, due to the same for nucleolar cells, and then the self-median (texture) difference, which I believe may be a relatively new choice for this particular classification problem.  Within-class standard deviations (smaller should be better) The most closely-clustered features happen to be the simple skewedness and mean distribution of pixel intensities, followed by the image size; unfortunately, these features are not particularly well-separated, but no matter - let SVM and knn theory do their jobs! To end off - so how do we know which classification algorithm is the best for a particular problem? Again, the answer is, we don't. However, dedicated researchers can give it a good go by trying many of them and checking the results, whether through their own private library of scripts, or more easily with integrated data mining software like Weka, and indeed the very best performers in complex classification problems often turn out to be very specialized ensemble methods. When that happens, it may feel like black magic, but hey, if it works, it works! The end-user's certainly not complaining, after all. It could end up working too well, such as the face detection cascade now spurring fears that Big Brother is watching everyone (it's far too late for such worries, actually), which has a NYU graduate student proposing methods to foil such systems. Unfortunately, it involves walking around looking like Lady Gaga on a bad day, and Mr. Paper Bag suggested that his longtime favourite solution could be less embarassing.  We Chinese do have built-in face recognition resistance (explained) (Source: funnyreign.com)
Plenty of leftover frozen pizza this week. And oh, got to the gym for the first time in a month. One Line  (Source: baidu.com) One Sentence A former secondary school principal has become the first victim of the borderline underaged vice scandal covered here previously, copping nine weeks for the oversight of not checking her identity card, as the judge sternly explained (graciously leaving space open for novel legal opinion when the next sweet young thing uses a fake IC). Never has the Yellow Ribbon Project been more warranted, but then again this sort of thing tends (rightly) to be mostly forgiven in short order, at least if local media personalities are any yardstick. Slightly annoyed at the ridiculousness of it all, I continued on with my perusal of the news, and came across an admission by no less than the Dalai Lama that he can be tempted by women, whereupon I along with Mr. Ham, who happened to be by my side, subconsciously stroked our chins and nodded meaningfully in unison. Sensible man, that Dalai Lama! (Unlike certain hypocritical religious leaders who shall stay unnamed - though he may in his revelation have outed himself as being not of Singaporean transport ministerial calibre) [N.B. But realistically, how much blame should be pegged on our newly-minted Minister for Transport? The seeds were sown long before he took office. And what actually can he do? Fine them peanuts, which will be more than made up from public funds anyway? Basically, he is being held responsible for something he realistically has next to no control over, which reflects the crazy state of the portfolio] Our joint estimation of the Dalai Lama having ratcheted up two notches, I poked about for more by His Holiness, and came across another of his quotes: "...And I also mentioned in case Dalai Lama's incarnation one female comes then must be very attractive female. So the very reason, you see more influence to others, an ugly female then may not much effective", at which point we slapped our hands simultaneously on the tabletop in unrestrained awe and exclaimed together, he speaks Truth! Has a more enlightened man ever lived? This has even inspired Mr. Ham to do some reading up on Tibetan Buddhism (though as the Dalai Lama said, don't become a Buddhist, the world doesn't need more Buddhists; do practice compassion, the world needs more compassion. Coming from a man who loves even George Bush [and therefore doubtless the rest of humanity], he really practices what he preaches), which I hope manages to counter Mr. Ham's exploitative Cult Leader tendencies somewhat. Ten Lines Well, you tell us that we're wrong And you tell us not to sing our song Nothing we can say will make you see You got a heart of stone You can never feel You say "oh I'm not afraid - it can't happen to me" ... Oh listen to the man in the liquor store Yelling "Anybody wanna drink before the war?" "Anybody wanna drink before the war?" "Anybody wanna drink before the war?" Wall Of Text Mr. Ham: *twiddling his paws* You know, after hearing what the Dalai Lama had to say, I think I've been a very bad hamster. Me: Glad that you realise that. So you're winding up your Cult Leader business? Mr. Ham: Well, I'm not going that far, having just attended a course on effective propaganda. You humans are so mean, saying bad things about the dashing Young Leader of the Glorious North Korea, when maybe all he needed was some love and encouragement - how different might the world have been had some stuck-up evaluator at the Vienna Academy of Art been more accepting? Such a talent, and he's not even the most famous Kim, it's enough to make the heart of a fellow expert break! He likes basketball - some NBA team should give him a tryout, they might get the Nobel Peace Prize. But, I can lend you my... uh, Esquire Pants' magic monocle. Me: Baby steps, I see. Fine by me. So what does it do? Mr. Ham: It allows you to access the little-known dimension located in what most people perceive as whitespace between lines of text. Extremely useful for lawyers, and anybody else, for that matter. Go on, try it out! Me: Alright, I'll surf to a random news report:
- academics - Since the organizers very kindly extended the deadline again (which I suspect is something of a custom in academic circles), I got another weekend to tinker with the cell data (not Excel!), and here's the overview:  I can't even begin to express my appreciation for datasets which can be quickly processed (which my main line of research, alas, is not), and since LIBSVM was blindingly fast too, I produced the above graph by adjusting the training and test set sizes. Recall that about a fortnight ago, I reported an accuracy of about 88%, obtained by splitting the entire set into disjoint training and test sets of (approximately) equal sizes, training an SVM model on the training set, computing the accuracy on the test set, and averaging for ten runs; in the above chart, the equivalent accuracy on the updated feature set can be obtained by following the line vertically upwards from 0.5 on the x-axis (i.e. training set size divided by the entire set size is 0.5) until it hits the red curve, and this returns an improved accuracy of 94.26%, averaged over a hundred runs this time (the accuracy for individual runs tends to be within ±3%) It can be observed that the larger the training set size in relation to the test set, the better the accuracy. This is not too surprising, since a larger training set intuitively contains more information, allowing better generalizations to be drawn. Note however that the cell classes are distinct enough that even with only a handful of samples (about five to ten) from each class, an overall accuracy of over 70% can still be obtained. The rightmost datapoint does not actually touch the line x=1, since it was obtained using the leave-one-out method - for each cell image, all the other images were used to train the SVM model, and since there is only one possible combination, no averaging is required. The accuracy achieved is 96.39%. It might then be instructive to consider the confusion matrix for the leave-one-out case (zeroes omitted):
From the matrix, six homogeneous cells were incorrectly classified as fine speckled cells (and so on). As can be seen, the bulk of the remaining confusion exists between homogeneous and fine speckled cells, which together account for 14 of the 26 errors. Just for completeness' sake, let us examine these intransigent lads (rescaled to identical size):  For the most part, I frankly can't see s**t, captain I suppose this shall have to do for now, having run out of time, though some of the misclassifications remained disappointing. This may also serve to illustrate that by far the most gains in such tasks (if not too complicated) are usually obtained at the beginning, and the final drops of incremental progress (see for instance the Netflix Prize) can be notoriously difficult to squeeze out. So, what features were used? Once again, there is nothing particularly devious here, with the final collection of 22 features chosen empirically. I suppose attempting to select them via a more principled procedure is possible too, but that will have to wait. The features used can roughly be put into seven categories:
If anybody's wondering why there's only one dark blob feature, blame my experiments (at the 50-50 split level) So how important are each of these categories of features? Running leave-one-out on various selected subsets (the average accuracies for equal-sized splits is generally a few points lower) of the full feature set gives:
*On hindsight, proper transformation of the images coupled with fairly good texture descriptors appears sufficient; however, as there were some particularly bad nucleolar cell misclassifications without the additional features, this did not sit very well with me, and therefore I opted for the larger feature set, which should still sit within the expected error range. Actually, I had also forgotten the basic mean feature of the statistical distribution category at one stage - that cost several percentage points. All this seems to imply that the problem might be less complex than I had expected - hopefully I have not made any dreadfully embarrassingly gross mistakes in the methodology, but in any case I'll find out in due time, when the official results on the real test set are released. Not too bad a learning experience for two weekends...
 Copyright © 2006-2012 GLYS. All Rights Reserved. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
