PA - Programming & Analysis - STRIKE 3!!! Useful feedback on failed exams in a problem.
Ok look.... I have a lot I would like to say about NCARB's testing format and what I perceive to be real problems with the way this test is structured, but, I don't want this thread to just be me ranting. Rather, my goal with this thread is to see how many of you feel the same way I do and HOPEFULLY either A) get enough of you involved in this thread that NCARB actually decides to change something, or B) get something useful from one of you that can help me on future tests.
Therefore, please engage in this thread. Just write something (Agree, Disagree, Somewhat Agree, <insert paragraph here>)
I want to hear from all of you out there if you are seeing/feeling the same way I am with regard to the way NCARB has structured these tests. Full disclosure: I am writing this after having just "preliminarily failed" the PA exam for the 3rd time, so if this article comes off like I am frustrated... I am. That said, I had a very respectful discussion with an NCARB representative over the phone today just to discuss the issues outlined below, and, well he was atleast respectful and sympathetic to what I was going through if nothing else.... he did provide me a email to some sort of helpful links for future study..... but, there really wasn't anything else...which is what I think is the issue as you will read. See below for a list of some of the barriers/challenges I feel are detrimental to the testing experience and not only my opportunity to succeed but yours as well. (Listed in order of most egregious)
- The policy that prevents test takers from using a piece of paper or pencil. Not even if it is provided by the testing organization. I am so tempted to rant on this but simply put... this is just wrong... the "whiteboard" is not an effective tool. No one, nowhere uses this archaic of a tool in our modernizing profession as a way of working out math problems. This tool is not a "feature" but a "hindrance" to anyone trying to succeed on these tests and become a licensed professional. I can't overstate enough how needless this policy is that NCARB can't simply allow test takers a blank piece of paper and a #2 pencil. The test is literally making the same thing in a clunky form that takes up screen real estate and becomes a real strategic barrier to taking the test when someone has to simply take the test in order to learn how to effectively navigate back and forth between reading the essays (otherwise known as problems) and then jotting down info on this whiteboard.
- The uselessness of these vague score reports. In short, I can see how someone might argue, "well at-least we get something back rather than just the word FAIL", but, to be honest that is a weak argument in my opinion and I just think these score reports are not in the slightest way helpful. NCARB reorganizing the words; evaluate, identify, prioritize, analyze, sustainability, environmental, etc. etc. into a bunch of similar sentences and compiling that into PDF does nothing to help narrow down the topic area(s), or sub-topic area(s), people are really struggling in. Those terms in the score report are so broad that I am sure one could read entire textbooks and not cover all the categories, sub-topics, and specialized topics that would fall under a term like; sustainability, environmental, qualitative & quantitative attributes of a site. Wow NCARB! Thanks for clearing it up for me.
- Score reports & semantic word games. This probably should be my #1 most egregious item since it is the one I am most upset / discouraged / hopeless about. Does anyone else feel like NCARB is needlessly confusing you and playing semantic tricks on you with vague terminology, poorly worded questions and even more poorly worded answers? As I am sure some of you know what I am talking about, you may feel like you have fully studied a content area and are a near master at it, however, then you get into the test and you're forced to decipher "hidden meaning", or, try and interpret the "underlying meaning" of specific descriptive words that NCARB has intentionally placed into the questions and answers as a way of INTENTIONALLY tripping you up! This is discouraging to me to say the least because I no longer feel like I am being tested about my knowledge of particular content but rather playing this odd game (a game that sometimes feels like a gamble) where I have to not only know the content but also try and figure out what the test writer means when they use certain wording...and then... "analyze", "evaluate", "synthesize" how this new wording changes my view of the content I am confident I know. But OK.. FINE NCARB.. I understand you need to make the tests a bit more difficult to see if testers can truly think through all the variables of a situation and conclude the "best" answer or "most appropriate solution" given certain variables. That's all well and good. I do see value in that. Fine, I will just have to play this semantic game and do my best if I want to get licensed, right? Challenge accepted. BUT, but what makes this issue so infuriating, what I do not find valuable and honestly leaves me feeling a bit helpless is, again, these super vague score reports that do nothing to help me as the test taker know if I am interpreting descriptive words about the subject matter correctly. To be very specific, there are questions where the wording of the question, or more likely the answer(s), IS the stumbling block and not my lack of knowledge on the content in question. I find myself wasting valuable testing time debating internally on what the test writer is trying to convey by using certain words over another. And, in many cases it feels like I am having to just take a gamble that my definition / interpretation of how a word is being used matches with the intent of the test writer's. Ok..so NCARB says, "Correct, that's how we made the test. That was intentional. So what?" And my response is, "How am I supposed to improve my test scores, correct my mistakes and get the answers correct on the 4th time I take this test if NCARB isn't going to tell me which problems I got correct, or more specifically which semantic word games I interpreted the underlying meanings correctly and which ones I didn't? I will never know if I am making the correct judgements and therefore I feel this is a major flaw in this testing format. I feel I am being tricked into second guessing information that I know to be true based on word games, word games that you won't tell me if I ever got them correct, so next time I take the test I could possibly know exponentially more about the content being asked....but....if I fail to interpret the semantics correctly and how those change / or don't change the correct answer then well... good luck gambling!
Ok so this did turn in to a bit of a rant. But I am bothered by this and I feel that if NCARB was more forthright in helping you understand what you got wrong and what you got correct there wouldn't be an issue. But it doesn't feel like they are doing that. I mean come on, I, like many of you, have sacrificed a lot of my money and time (including our family's time) to study for these tests and work to become a licensed professional, and well, I simply believe that NCARB's practice of withholding scoring information, making everything vague and simply not providing USEFUL feedback is the ultimate issue here that continues to cost us testers more time and more money. The wording in the tests is a problem, but, the lack of real feedback is the real issue.
I agree whole heartedly about the whiteboard. We are trying to become architects and in the most basic sense we should be sketching and note taking. I understand the problem with giving testers at home paper, fine… I don’t know what that solution is exactly but now that COVID is less of a threat I don’t care if that option goes away if it means we get paper back! My bosses were horrified to learn we were not allowed paper - they both said there is already becoming a big disconnect between sketching/hand drawing and what we do. I agree.
The others I’m not sure I agree as fully. There is such a fine line between giving feedback and letting test answers get out… I have one fail under my belt and I did find the report at least somewhat helpful to focus for my next attempt. That being said- PA covers way more so I may feel differently if that was the fail.
I will also add I left CE (pass) feeling annoyed that I didn’t know the answers to the questions I knew I got wrong. It seems like it would be important to make sure we understand our mistakes so we don’t make them “in real life”. But again, I understand that fine line. It ducks though.
For the wording and such, have you taken NCARBs practice exam? Maybe that would help you get the hang of how questions are being presented?
Thanks for your feedback! Thus far I have not taken any of the practice exams by NCARB, but I definitely think that is a good suggestion and something I will need to considered working in to my studies.
So far I have been using AmberBooks and the Ballast 2nd addition book by PPI. Since using AmberBooks I am 2 for 3 on passing. I really have liked AmberBooks and will continue to use that as my primary study material. The content, the flash cards, the exams, and Michael’s embedded humor within all of his nice and lengthy explanations are reasons I can’t stop using this program.
I am not really sure yet how to study for the semantics. Perhaps those practice exams will help but I honestly think this is a problem with the test format that distracts from the ultimate goal of simply knowing the content. I totally agree with your point too about knowing your mistakes so you don’t go out and make them in the real world.
I highly reccomend looking at the practice test from NCARB. You can go through and get the answers question by question. I think it would be helpful for you to be able to click through and even if you just did did the ones you see those tricky phrases (best practice, evaluate ect) you could just do those. Each question does have feedback so it would let you see what they are expecting. My experience (3 passes relying heavily on NCARBS practice test and main sources from the matrix) has been that the real exam feels exactly like the practice.
Adding - the practice exams also have a final scorecard with percentages that shows which questions went into which categories and if you would have passed. That will probably help you see where to focus from the categories.
Agreed... additionally, this exam had next to NOTHING that was reference on the practice exams. Why provide practice exams if they are just a waste of time?? Days wasted studying for the exam - what a joke. NCARB you need to hear us all & update some things. This is an insult to my training/experience & a real cut into the money I could be making in place of taking weeks upon weeks to study for an exam that is impossible to pass.
Vicky Cimmino I feel your pain. I felt the same way after having studied multiple times for this same exam, each time covering more and more content while also reviewing content I thought I already knew. And somehow, after studying (3) times for this test, my scores aren't getting better but worse. Some areas were better but it's crazy how on the first exam I scored really strong in one area, then on the 3rd exam I scored extremely low in the same area. This system seems flawed because of this:
1) If you pass the exam - NCARB doesn't tell you what you got right and wrong. In fact, they don't give you ANY report at all.
2) If you fail the exam - NCARB doesn't tell you what you got right and wrong. (the reports they provide are not that helpful because it doesn't help me narrow my studies towards actual content I got wrong. Instead, you are forced to study everything that could possibly fall underneath their broad topic areas)
3) Therefore, you won't truly ever know what questions you got right and which ones you got wrong. Hence, you will never know after somehow passing all the exams which subjects you ACTUALLY know well vs the ones that maybe you just barely passed in.
BUT.... all that complaining aside.... I have purchased my 4th seat and am back on the saddle. I am going to try and take PA after I take PPD and PDD and see if my overlap of studying in those areas don't help me to pass PA.
Good luck. Don't give up!
I'm with you and supplied my Rant on this Horrible test a few months ago. I'm now attempting my 5th retake. I have taken this so many time that I actually recognized the sample test questions from when I actually took them on the official test. I have passed almost all the over 5.0 test in my first attempt, this is the only one that repeatedly Fs me and now its my very last exam. I agree the questions and resources are needlessly vague and at the same time so specific. I also hate the pick 3 and pick 4 and it not being scaled if you get 3/4 and miss the last one when most of them are applicable.
I enjoyed your rant and whole heartedly agree with your position.
Branden Adams it is nice to know I am not alone in these struggles. Man, I feel for you and that 5th retake. To your point on the PICK 3s or PICK 4s, you are spot on. Again, I feel like I am beating a dead horse here when I say that the "feedback" NCARB says they provide is a joke. You and I will never know if we got 3-out-of-4 answers correct, or 2-out-of-four correct, or if we got all of them correct. So what's the point! And to be honest true feedback would need to go further than just telling me which answers I got correct and which ones I got wrong, for example, just seeing a scantron with correct answers and incorrect answers would be no more helpful to us than what they are doing now, IF they didn't also provide the actual question and answers so you could reflect on the words, the problem, etc. to know how and why you got it wrong. Maybe you just didn't know the material, maybe you knew the material but second guessed yourself due to wording of the answers and selected 1-out-of-4 wrong, maybe there is a situation (like on these problems that ask you to drag/drop the red X's into place) where you knew how to answer the question but you simply placed the red X in a location that was somehow outside of the acceptable area. Like to be a bit more specific, let's say you knew that the depth of the stair treads were not long enough for height of the risers. Therefore you noticed the dimension of the overall stair was too short. Do you place the X on the actual stair treads, the stair risers, or do you place the X on the overall stair dimension string since you know that each of the stair treads need to be longer? The point is.... IDK.... and you don't know.... and who knows? NCARB knows but I won't ever know if I actually understood the problem and simply misplaced the little red X in a location that wasn't "acceptable" to NCARB or if I just simply didn't know the content.
So this is a separate story, but, as mentioned in my original post I spoke with an NCARB rep. And as I said before, he was really nice and professional and I appreciated his willingness to try and help me even though the amount of help he truly could provide was extremely limited given the current restrictions around testing feedback. But he was a nice guy and he was trying to come up with alternative options for me and said the following, "well sir, if want to pay a non-refundable $100 to NCARB we will allow you to dispute a question." And while again, I am grateful for this guy genuinely trying to come up with options, I was just laughing internally at the whole concept of giving NCARB $100 of my money to dispute (1) question (mind you... YOU CAN'T SEE WHAT QUESTIONS YOU GOT WRONG... SO HOW WOULD I KNOW FIRST WHICH QUESTION TO CHALLENGE and SECOND, EVEN IF I WERE SOMEHOW LUCKY ENOUGH TO GET NCARB TO SEE IT MY WAY AND COUNT THAT (1) PROBLEM CORRECT... WHAT DOES IT MATTER IF I GOT A BUNCH OF OTHER PROBLEMS WRONG? HAHA like it is completely laughable at its face that this is an option for "disputing your test results" because you as the test taker have no clue; what questions you got wrong, how many you got wrong, or the amount of grey area given in a problem that would give you understanding whether it worth your time AND MONEY to challenge a problem. Sounds like a sure scam to me and a great way to throw away $100. Now, perhaps I am misunderstanding exactly how that process works, and if I do, I apologize for my ranting and overall mischaracterization of NCARB and their staff..... IF I understand it wrong.
I mean has anyone out there ever actually done this option? Has anyone out there actually ever had success disputing a question/answer? If so, what was the process like? To me it seems like a complete gamble based on what I said above.
Ok so... then the second thing this NCARB rep said was, "Well sir you could also consider paying an additional $200 to have a panel of people review some things with you." And, to this my ears initially perked up! I was a bit hopeful of this and I will be honest, I still don't know if I completely understand exactly how this process works. But, if I understood this NCARB guy correctly this panel of people (and maybe it is just one person) would review your test for you, see what you got wrong and got right, then be able to meet in person to discuss (IN GENERAL VAGUE TERMS) the areas you may be weak in vs stronger in.
IDK... I will admit that I don't know enough about that second option but again it seems like a waste of $200 because the professional meeting with you can not discuss any portion of your test directly but only give general guidance on what to study, where your strengths are, etc.
So Branden Adams if you want to give NCARB $300 you could dispute a question (any question) and possibly meet with a professional who could give you some more vague direction.
Just pay for another seat and take the retest for $235 or whatever the cost is. When I look at those "options".... they don't really feel like "options" to me. One is a gamble with a high chance of failure, and, the other is well another retake.
IDK. I the middle of all my pessimism I really don't want to leave this rant all negative. At least that guy at NCARB was nice and respectful (like I really was impressed with that given how downtrodden and frustrated I was that day I failed for the 3rd time), and, well I do appreciate some of the changes NCARB has made to make testing easier (for example, being able to reschedule tests FOR FREE). And though I didn't ever choose to use the option of online proctoring at home, I was grateful for NCARB providing that option during COVID. There for a while I thought that might be an option I might take due to the craziness in the world. So... it's not all bad and there are some positive things NCARB has done.
Josh, after my last attempt is was MAD! My BS Score report said I did worse than my previous attempt (which was what I could understand an almost passing attempt) and afterwards I did go down the rabbit hole of do I pay to get this reviewed. Finally calmed down and figure it was a less stressful to simply attempt a retake.
I have my hopefully final retake this upcoming week and have been trying to take meticulous notes on the process so hopefully I can pass out and help the rest get beyond this nightmare.
I do agree that the new rescheduling protocols are fantastic.
Thank you, Joshua Cristy, for your post! I subscribe to every word. I failed my 4th attempt with PA. It is my last exam. I was so sure, that if I study for PPD (passed from the first attempt) & PDD (from the second), I will nail this one down. Nope. This exam is not about the knowledge or practice. I don't know what even to study now. I found out also that they have some mistakes in the test... at least it seems so.
Sometimes I think that 15 years in the architectural practice makes passing this test even more difficult as I consider a lot of information from real life experience, not only from the study resources.. So devastated...
Victoria Maliuta I am so sorry for you and I feel your pain. Gosh I sympathize with the pain of failing it again...Just know that you aren't alone! As you can read in this one thread, there are lots of people that feel the same way. In fact, it isn't just my post that has gained traction, there are many other threads started by other examinees who have voiced similar complaints on other threads.
Can you share what all you have done to prepare in each of these 4 attempts, like what study resources or practice exams you've used?
Personally, every time I have spent any amount of time utilizing PPI to study for the PA exam I have failed. How many time would that be? 3 Times. The third time I also used Amber Books but part of me felt like there might be some value in continuing to read the PPI Ballasts Book for the PA, PPD chapters. And IDK, I have heard of others who found PPI to be sufficient for them, literally like (1) that I can think of, but from personal experience I have failed 3 out of 3 times I have spent time with PPI. Amber Books on the other hand has been really good thus far. I have PPD and PDD coming up in early February that I am using Amber Books to study for. I am hoping that Amber Books alone + NCARB Practice Exams will be enough to get me across the finish line on PPD & PDD. My recommendation would be to spend the money and go with Michael's Amber Books. While Michael Ermann and Amberbooks could really benefit from some updated videos where they fix a few of the typos, errors in calculations, and other issues prior amberbooks subscribers have posted months and years ago.... I still, thus far, have greatly enjoyed Michael Ermann's work and depth of knowledge of the subject matter. I definitely feel like I am growing in understanding and deepening my knowledge of architecture due to @Michael Ermann's (and AmberBooks Team) great work.
If you are wondering where to go next after failing 4 times that would be my recommendation. I would say spend a good amount of time going over the primary content as Michael lays out, but there is A LOT of good information and great links to additional information within the flashcards and practice exams.
Victoria Maliuta I understand the struggle with wanting to spend a couple hundred dollars to study for one more test, but, if what you have been doing for some reason isn't working perhaps it's time to try something different. AmberBooks is packaged and advertised as a wholistic education on ever test / subject matter. Michael has provided some significant reasoning for why he believes everyone should watch all of the videos and then sit down and take the tests all together. That said, I believe Michael and the Amberbooks team are wise enough to know that even though they strongly recommend you should watch all the videos (due to the vast amount of overlap in the content you will see on any of the PA, PPD, PDD tests especially), they have provided a "study matrix" guide at the beginning of the program that allows people who would prefer to not take Michael's advice to watch the select sections that pertain to their particular test. Also, if you hadn't done this already, you should visit the www.nps.gov to read through the Historical Treatment types, standards, and guidelines. I have found understanding historical renovation types/methods to be pretty key to the PA exam.
Vicky Cimmino It is my understanding, as explained to me by the NCARB rep I spoke with a few weeks back, that the NCARB community IS the place for us to find answers to our questions, voice our concerns, and to generate helpful conversation for NCARB to review and hopefully generating change. Also, it should be expected that there is likely nothing any of us will see change in the timeframe we are taking our tests (Lord willing we all pass in a year or two). Even if there was an issue brought up in this forum, or any other forum, that NCARB found worthwhile to correct, it most likely would not be implemented soon enough for any of us to benefit from that change. BUT... again if there is anything constructive here it is possible it could help out others down the line should NCARB choose to act on them. Therefore, to be honest, this NCARB community really does kind of feel like a place to come do a couple things: 1) Get quality feedback from other examinees on what materials to study 2) Connect with other examinees to setup study groups or simply make connections 3) And as "un-sexy" as it is, sometimes simply to vent our struggles / complaints. I understand that sharing complaints / struggles is a bit of a downer and who wants to read that stuff anyways. Our world is filled with too much pessimism and complaining as it is - hence the reason I tried in some of my later posts to try and point out a few areas I though NCARB was doing a good job. However, having a place for people to post their concerns, struggles, etc. I have found valuable even if my only benefit was to message with other individuals going through the same struggles. Finally, I do think it is valuable for people to post if they have legitimate concerns with the test or the procedures of taking the test, how else is change supposed to happen if constructive criticism isn't provided. If a lot of people are finding there to be issues with a test, where else are people to go to voice those concerns? According to the NCARB person I talked to, this NCARB community was the place.
Finally Finally, in the spirit of reflecting on some of the positives I will say I have always appreciated NCARB's user interface user experience (UIUX). The team that works on their user interface, graphics, pdfs, etc. does a great job. I feel like information is easy to find, easy to read, color schemes are consistent across different documents. Very enjoyable platform to navigate and find what I need.
The information is valuable on here. And I do understand that any updates would probably be in effect long after any of us needed the help. I was simply stating that we all should "go to the source" individually to possibly see some change.
This forums are valuable but these exams shouldn't be THIS difficult, especially with someone like myself who has just over 20 years in this industry. I do PA daily, there is NO way that I should have failed this exam. Honestly, this should have been the easiest for me & for all of us quite honestly.
Vicky Cimmino I agree that PA has been surprisingly the hardest of all my tests thus far when I truly thought I would do well on this test the first time (especially by the second time). Never would've dreamed that I could fail the same test (3) times. Yea it has got to be frustrating having the amount of experience you do in real world practice only to find out that all of this real world experience that has likely served you well (could be - maybe not entirely the case) could be working against you in these tests. I feel the same way you do with less experience. Anyways, I can't help but fall back on my original complaint and say.... what are we gunna do? If you and I are never given direct feedback on what we got wrong /vs/ right I feel like we are simply not in a place where we can do anything other than take the test again, do our best, and hope we hit a "blackjack"! I use the gambling references because if you know you failed a test, but you don't know which questions you got right, which ones you got wrong, which ones you guessed out originally (meaning you didn't actually know the content) but got right, versus the ones you guessed at and got wrong, if you don't know these results then every time you take the test you are forced to double think whether your interpretation of subject matter was correct the first time, or, if you ought to change your approach the second time you take the test. By the time you are in your 3rd or 4th retake I am sure you feel similar to many of us where you have studied the material enough to understand a large body of content but you are constantly second guessing yourself and any prior knowledge you brought into your previous exams because well..... who knows???? did you get it right last time?? Did you get it right the first time?? How about the 3rd time?? The 4th time?? Nobody knows, except NCARB. This isn't right.
Josh, Fifth time was a charm. Just got my Formal Passed result and now I'm all done.
Ill make a post of areas I focused and what I found to be helpful. but honestly taking it 5 time in the course of two years I probably built up a solid knowledge base.
keep up the good fight! and Rob Schneider quote helped.
Branden Adams CONGRATULATIONS! Yes please share any thoughts you have on what study areas you felt got you over the finish line. Man, I am sure with the amount of studying you had to do on this test you could teach the subject and recoup those gambling losses in no time!
Well congrats on being done!. I've got (3) more. PPD, PDD, and PA (4th try). All in February.
I'm happy that I read this thread. I'm fully agree with
1. Whiteboard don't do much of the job. I bought a wide monitor in order to use the space on the monitor. It helped a little but not much. My suggestion to NCARB (if someone is reading it) Please add a option to have digital pen or something. Approve some WAcom or XPEN devices and we pay for them. Would be much cheaper that to pay for a new exam, especially when you have a ton of information to calculate (area, budgets etc)
2. Wording - I'm not native English speaker so I've always thought that it's just me. Obviously not. And in addition to wording, as and emerging architect I think that some of the questions don't have one definitive answer. Sometimes the answer is depend of other information that is not provided and the correct answer is questionable and debatable. I think those answers are about 30%.
3. Feedback. (same as exam objective) not clear what question go to which category.
Hopefully the exam will change in the wall to test you knowledge, not just brainstorming and adaptation ability.
Here is my POV as a Licensed Landscape Archtitect
The site stuff, urban planning stuff, and zoning stuff is absolute trash. It was clearly written by people who have no clue how site work, urban planning, and infrastructure is really done. It comes off to me as technically illiterate people who resorted to pulling random passages out of a handful of site engineering manuals.
Pick which stormwater treatment is best for a poet living in a cottage built on land reclaimed from the sea.
Then you pick (D). but its wrong because a poet would prefer the "softer form" of a swale. (A)
But in reality, in the real world all those words are synonyms for the same thing and there is no official professional use.
A better way would have been to have a table of 7 different stormwater treatments with columns for TSS, N, P, and Heavy Metals and ask you which one was best if the goal is water quality. Then you would do what every civil engineer and LA does, look up the highest number in the TSS column and pick that one. That shows real understand and analyze and its objective and reproducible.
The correct answer to like 90% of PA division questions in the REAL world are: "wait for the civil to respond"
The white board is total trash. NCARB needs to either 1. allow paper, 2. increase the time limit, 3. add WACOM tablets. In addition, you need to be able to DOCK the calculation and white board so you have tidy well organized work space. The test UX design is total trash, it actually makes it HARDER not easier to pass the test.
yeah so anyway PA is total crapshoot the way it stands currently. it in no way tests your authentic mastery of planning and site design knowledge.
How did it go Joshua Cristy? …. Asking bc I’ve failed twice, same thoughts about the test. I want to know what finally clicked when passing after failed attempts ? ….I’m Trying to figure out what to do differently for my 3rd try. This is my last exam and the only one I’ve failed.
Please sign in to leave a comment.